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BROADBAKD MID-NETWOKR SERVER 

The present invention relates to internetworked 
communication systems, and especially (but not 
exclusively) to a highly scalable, broadband mid-network, 
server for performing mid-network processing functions 

5 including routing functions, per user processing, 

encryption, bandwidth distribution and traffic shaping. 
Background and Siommary of the Invention 

As bandwidths within the core of the Internet increase, 
there is an increasing trend towards using the Internet 

10 Protocol ("IP") as the core network layer protocol for all 
kinds of traffic, including voice, video and data. 

Historically, quality of service on the Internet has been 
what is called "best effort." That is, the network attempts 
to transport as much traffic as possible, but if there is 

15 insufficient capacity to handle the traffic, all connections 
are equally likely to be influenced by congestion. Thus, 
"best effort" in^lies that the Internet provides only one 
class of service to any connection, and that all connections 
are handled equally with no priority. 
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In the case of traditional Internet applications, this 
approach was often sufficient. However, the intrinsic 
potential of the Internet is considerably greater, and 

i ncludes new multimedia and interactive a p plications > Voice 

5 over IP ("VoIP") and Real Time Video are envisioned to be two 
significant applications for propelling Internet growth to the 
next level. VoIP can be defined as the ability to make 
_ telephone calls and_ send faxes over_IP networks. The benefits 
of this technology are cost reduction, simplification, 

10 consolidation and advanced applications such as shared screens 
or whiteboarding which combine voice and data. Real Time 
Video is a direct- to-user'' technique in which a video signal 
is transmitted to the client device and presentation of the 
video begins after a short delay for data buffering, and 

15 eliminates the need for significant client-site storage 
capacity- It is also expected to become popular with 
businesses. Related to this is webconferencing, which 
requires high bandwidth since it is a continuous transfer of 
image information together with voice transfer. 

20 Webconferencing also requires real time traffic handling 
because it is usually inqplemented as an interactive 
application - 

All of these new applications will generally require 
significant bandwidth and/or reduced latencies. Bandwidth is 

25 the critical factor when large amounts of information must be 
transferred within a reasonable time period. Latency is the 
minimum time elapsed between requesting and receiving data and 
is important in real-time and interactive applications such as 
webconferencing and telecommuting. Presently, most 

30 telecommuters depend upon analog modems with limited bandwidth 
and significant latency for dial-up connectivity. Even for 
today's applications, dialup connectivity is often inadequate. 

There are competing "last mile" technologies today which 
provide transport services to the user for delivering packets 

35 to the "edge" of the Internet. To complete the communication, 
the packets need to be formatted to allow them to enter the 
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Internet cloud and find their way to their respective 
destinations. The emergence of supporting protocols for new 
applications and the growth spurt in nuanber of users and the 

rec f uired bandwidth per user results in a very d ynam ic acc ess 

5 environment - 

The following is a sxunmary of observations that pertain 
to an ideal mid-network point within the Internet: 
__ ^ _ In order, .to_accoinmodate _a .variety of source_packets.,_ 

all the requisite protocols must be efficiently 
10 supported. 

♦ Virtual Private Network services allow a private 
network to be configured within a public network. 
This is one of the drivers for Internet access amongst 
businesses. To allow Virtual Private Networks to 

15 coexist on the public Internet, and to encourage 

business use of the Internet, great care must be taken 
with respect to security and authentication issues, 
and tunneling protocols such as L2TP and IPSec must be 
efficiently supported. 

20 • The number of subscribers handled by one system and 

the different qualities of service provided will make 
service provider administration more complex. To make 
provisioning of broadband access more attractive to 
service providers, subscriber management and usage 

25 accounting must be simplified, and differentiated 

services must be provided. 

♦ Broadband makes it possible to provide different 
amounts of bandwidth to users and to smaller Internet 
Service Providers. To make wholesaling of IP 

30 connectivity possible, and to simplify service and 

repair functions, the ability to support multiple 
service providers with one mid-network server must be 
provided . 

♦ A large number of connections are serviced with a 
35 broadband mid-network server. In order to ensure that 

service is not interrupted, the broadband server must 
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have very high availability. Such availability is 
also required for mission-critical business 
applications . 

'Central" of f ice— co^locatron—space—is liiaited: — To 

5 conserve this space, large connection densities must 

be provided. 

• When subscribers are allowed access at high speeds, it 
~ i^ p~os'sible "for a limited numberuf "users' demanding' ' 

disproportionate amounts of bandwidth to disrupt 
10 service for other customers. To ensure that large 

traffic bursts do not overload small client buffers, 
and to ensure that service providers and customers are 
treated fairly, traffic shaping must be provided. 

• To enable new value-added services, large bandwidths 
15 and low latencies are critical. 

In order to solve these and other needs in the art, the 
inventors hereof have succeeded at designing and developing a 
broadband mid-network server that, in the most preferred 
embodiment, satisfies all of the requirements described above. 

20 This inventive server provides reliable, secure, fast, 

flexible, high -bandwidth, and easily managed access to the 
Internet so as to accommodate all current Internet services 
including email, file transfer, web surfing and e-comraerce, as 
well as the new value added services such as VoIP and Real 

25 Time Video. To meet these requirements, the broadband mid- 
network server of the present invention has been designed to 
scale not only in bandwidth, but also in processing power and 
state space. In the preferred embodiment, the architecture 
allows a service provider to configure the cards chosen for 

30 use in the available chassis space to suit his particular 
application. For example, to maximize processing power, a 
service provider could increase the number of IPE cards at the 
expense of a fewer number of line cards; as few as one line 
card. In the case of one line card, the maximum amount of 

35 processing power would be available to a service provider. IN 
the preferred embodiment described in detail below, this 
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configuration would provide 240 processors and 39 gigabytes of 
memory. This would allow for a greater nvnnber and complexity 
of value added services which require more processing power. 

-A-l-ternatdveLy^-a-greater_number_of_line_cards_.could^ 

5 selected for use in a chassis which would be desirable to 

handle greater traffic and throughput at the expense of fewer 
value added services. 

_ — The- high- bandwidth _co re _rQuters_ that are currently jinder_ 

development by third parties are optimized for perfoming 
10 large numbers of fast routing lookups, but are not expected to 
provide generalized and flexible computing power for 
supporting the 5\ibstantial amount of processing needed for, 
among other things, per user and per packet processing. In 
contrast, the broadband mid-network server of the present 
15 invention includes the ability to distribute traffic across a 
ntmiber of Internet processing engines and, more specifically, 
across a nvimber of protocol processing units provided in each 
engine (the bandwidth to which can be coordinated) , to provide 
conqpute power and state space required for performing per user 
20 processing for a large number of users - 

One important feature of the present invention is a 
unique architectural philosophy, which provides that 
processing, be performed as close to the physical layer as 
warranted by considerations of flexibility, cost and 
25 con4>lexity. This architectural philosophy maintains 
balance between two kinds of processing which are 
important to scaling bandwidth with value-added services 
in broadband networks: time-consuming, repetitive 
processing; and flexible processing which must be easy to 
30 program by third parties. The need for considerable 

time-consuming repetitive processing, which has proved to 
create a bottleneck in the processor -based servers of the 
prior art, is addressed by the inventive architecture 
through specialized hardware, and results in dramatic 
35 increases in speed and decreases in delay. The need for 
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flexible, easy to use, computing power to enable service 
providers to scale with value-added services is addressed 
by the inventive architecture preferably through the 
provision of high-performance general purpose processors 
5 which are paralleled and which can be scaled to a 

virtually limitless degree. Alternatively, network 
. _ prpcessprs ordigita]^ signal processors or any other 
programmable processor could be utilized as well. 
Accordingly, the . broadband mid-^network server of the 
10 present invention provides a system that is currently 

unrivalled in performance and which can become the prime 
mover of Internet services such as managed, secure VPNs, 
Voice over IP and Real Time Video. 

While some of the principal features and advantages 
15 of the present invention have been described above, a 
greater and more thorough appreciation of the invention 
may be attained by referring to the drawings and the 
detailed description of the preferred embodiments which 
follow. 

20 Brief Description of the Drawings 

Fig. 1 illustrates a single shelf broadband mid- 
network server according to one embodiment of the present 
invention; 

Fig. 2 is a functional block diagram of the 
25 preferred server shown in Fig. 1; 

Fig. 3 is a functional block diagram of an exemplary 
line card shown in Figs. 1 and 2; 

Fig. 4 is a functional block diagram of an exemplary 
IPE card shown in Figs. 1 and 2; 
30 Fig. 5 illustrates routed distribution to an IPE 

card; ' 

Fig. 6 illustrates the processing flow on an IPE 

card; 
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Fig, 7 illustrates a protocol processing platform 
according to the present invention; 

Fig. 8 is a functional block diagram of an exemplary 
buffer access controller; 
5 Fig. 9 illustrates the format of a cell received at 

an input to a BAG from a PIC; 

Fig. 10 is a functional block diagram of a preferred 
packet manager; 

Fig. 11 is an illustration of the deployment of a 
10 broad-band mid-network server at a Service Provider POP; 

Fig. 12 is an illustraion of the different kinds of 
links an ISP may want on a secure segment; 

Fig. 13 is an illustration of the system wide 
bandwidth distribution functions; 
15 Fig. 14 is an illustration of the multi-level 

policing and multi-level shaping that occurs in the 
system; 

Fig. 15 is an illustration of router distribution, 
two level policing, routing and two level shaping; 
20 Fig. 16 is a functional block diagram of a preferred 

packet inspector; 

Fig. 17 is an illustration of the preferred 
. Distributor Flow Unit; and 

Fig. 18 is a summary of the highlights of the DFU. 
25 Detailed Description of the Preferred Embodiments 

The mid-network processor of the present invention 
is preferably implemented in a single shelf system as 
shown generally in Fig. 1, and is indicated generally by 
reference character 300. As shown in Fig. 1, the mid- 
30 network processor 300 is provided with a number of 

physical connection ("PHY") cards 302-316 through which 
packets may enter and exit the mid-network processor 300 
according to a particular communication protocol, as is 
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known in the art. For the preferred embodiment 
illustrated in Fig. 1, the mid-network processor 300 

supports the POS, ATM, and Gigabit Ethernet layer two 

protocols, although the mid-network processor may Iceadi'ly 
5 be configured to support additional protocols, as will be 
apparent. The PHY cards 302-316 are each associated with 

- - ,line cards 322^336, respectively, as shown in Fig. 1. As 
is well known in the art, each PHY card is media 
specific. In other words, each PHY card is provided with 

10 connectors and other components necessary to interface 
with the communication media connected thereto, and over 
which packets enter and exit the PHY card. Each line 
card is configured to process packets of the type 
received from its associated PHY card, as explained more 

15 fully below.. 

'The preferred mid-network processor 300 shown in 
Fig. 1 is also provided with a number of Internet 
Processing Engine ("IPE") cards 340-354, as well as two 
flash memory modules 360, 362 and four switch fabric 

20 modules 364-368. As appreciated by those skilled in the 
art, the. number of switch fabric cards required is a 
function of the switch fabric card design as well as the 
desired redundancy overall performance. Fig. 1 also 
illustrates a midplane 370 that is provided for 

25 interconnecting the various cards described above. The 
preferred mid-network processor 300 utilizes a card-based 
approach to facilitate maintenance and expansion of the 
mid-network processor 300, as necessary, but this is 
clearly not a limitation of the present invention. 

30 The manner in which packets are processed by the 

preferred mid-network processor 300 will now be described 
with reference to Fig. 2, which is a functional block 
diagram of the preferred mid-network processor 300 shown 
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In Fig. 1 (although, to simplify the illustration. Fig. 2 
does not show the PHY cards 310-316, the line cards 330- 
336 and the IPB cards 346-354 shown in Fig. 1). Packets 
enter the mid-network processor~"300~vi~a~the— PHY— cardsv — as 
5 is known in the art. Each PHY card then delivers its 
packets to its associated line card through the midplane 

370. _ After performing initial processing of the packet, 

the line card deliviers the packet again through the" 
midplane to the switch fabric which, in turn, delivers 

10 the packet to one of the IPE cards for performing certain 
mid-network processing functionis, such as routing 
functions, per user processing, encryption, and bandwidth 
distribution. After performing mid-network processing 
for the packet delivered thereto, the IPE card sends the 

15 packet back into the switch fabric, typically for 

delivery to one of the line cards for some additional 
processing before allowing the packet to exit the mid- 
network processor 300 through one of the PHY cards. In 
some cases, depending upon how the mid-network processor 

20 of the present invention is implemented, a single IPE 
card may be insufficient to complete the necessary mid- 
network processing functions for a packet delivered 
thereto. In this case, upon performing some processing, 
the IPE card will deliver the packet to another IPE card 

25 (rather than to one of the line cards) via the switch 
fabric for further processing. Thus, although a packet 
will typically be processed by only one IPE card, it is 
possible to process a packet in multiple IPE cards, if 
necessary. 

30 In this preferred embodiment, all of the line cards 

contain identical hardware, but are independently 
programmable. Likewise, all of the IPE cards contain 
identical hardware, but are independently programmable. 
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This contributes to the scalability and elegantly simple 
design of the preferred mid-network processor 300. 
Additional processing power can be provided to the mid- 
network processor by simply adding additionajrTPE~*carasT — 

5 Similarly, additional users can be supported by the mid- 
network processor 300 by adding additional line cards and 

- — ^P-HY- cards, _ and _perhaps_ additional IPE cards to provide 
additional processing fpr the newly added users, if 
necessary. 

10 The flash memory cards are provided for storing 

configuration data used by the IPE cards during system 
initialization. 

* Note that, as used herein, the term "packet" refers 
to any type of packet that enters or exits the mid- 
15 network processor 300, including packets input to the 
mid-network processor 300 in the form of cells (such as 
ATM cells) via an interleaved or non- interleaved cell 
stream. 

In general, each line card used in the preferred 
20 mid-network processor 300 performs a number of functions. 
Initially, the line card converts packets, (possibly of 
varying lengths) delivered thereto into fixed length 
cells. In this preferred embodiment, each line card 
converts input packets (including packets represented by 
25 individual cells) into 64 byte cells. The line card then 
examines the stream of fixed length cells "on the fly" to 
obtain important control information, including the 
protocol encapsulation sequence for each packet and those 
portions of the packet which should be captured for 
30 processing. This control information is then used on the 
line card to reassemble the packet, and to format the 
reassembled packet into one of a limited number of 
protocol types that are supported by the IPE cards. 
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Thus, while any given line card can be configured to 
support packets having a number of protocol layers and 
protocol encapsulation sequences, the line card is 
conf igujSed^to~~conTC3rF^ — - 
5 encapsulated packets (or, stated another way, into 
packets having an encapsulation sequence of one) of a 
type that is supported by each of the IPE cards. The 
line card then sends the reassembled and formatted packet 
into the switch fabric (in the form of contiguous fixed 
10 length cells) for delivery to one of the IPE cards that 
was designated by the line card for further processing 
that particular packet. 

Although the fixed length cells which comprise a 
packet are arranged back to back when the packet is 
15 delivered to the switch fabric by a line card, the cells 
may become interleaved with other cells destined for the 
same IPE card during the course of traversing the switch 
fabric. As a result, the cell stream provided by the 
switch fabric to any given IPE card may be an interleaved 
20 cell stream- Thus, the IPE card will first examine this 
cell stream "on the fly" (much like the cell stream 
examination conducted by the line cards, explained above) 
to ascertain important control information. The IPE card 
then processes this control information to perform 
25 routing look-ups and other mid-network processing 

functions for each packet delivered thereto. The control 
information is also used by the IPE card to reassemble 
each packet, and to format each packet according to the 
packet's destination interface. The IPE card then sends 
30 the reassembled and formatted packet back into the switch 
fabric in the form of contiguous fixed length cells for 
delivery to one of the line cards (or for delivery to 
another IPE card, in the case where additional mid- 
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network processing functions must be performed for the 
packet in question) . 

As noted above, although the cells of any given 

packeTflnay" enterrt 

5 arrangement, these cells may become interleaved with 
other cells during the course of traversing the switch 
fabric. Thus, the stream of cells provided by the switch 
fabric to any given line card may be an interleaved ceil 
stream. Accordingly, a line card will first examine this 

10 cell stream "on the fly" to ascertain important control 
information that will be used primarily to reassemble 
packets, and to format the reassembled packets for their 
destination interfaces. Additional processing of 
outbound packets is also conducted on the line card for 

15 PHY scheduling and bandwidth distribution purposes. 

. While the preferred mid-network processor 300 of the 
present invention has been described as delivering 
packets from a line card to an IPE card and then back to 
a line card (or to one or more additional IPE cards) , the 

20 mid-network processor 300 can also be configured to route 
cells arriving over an ATM interface on one line card 
through the switch fabric and directly to another line 
card ATM interface, and can therefore function as an ATM 
switch. 

25 Fig. 3 illustrates an exemplary line card 380 used 

in the preferred mid-network processor 300 of the present 
invention. As shown therein, the line card 380 
preferably includes an ingress side (i.e., the left half 
of Fig. 3) and an egress side (i.e., the right half of 

30 Fig. 3) . When packets are provided to the ingress side 
of the line card from the line card's associated PHY 
card, the packets are first provided to a packet 
inspector chip ("PIC") 400 which converts the packets 
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(which may already be represented by individual cells 
such as ATM cells) into fixed length cells. In this 
preferred embodiment, the fixed length cells are 64 byte 

cel-ls-that— are-8-bytes-wide-and-8-bytes— long^ — Thus,- -a 

5 "cell time," in the context of cells propagating within 
the preferred mid-network processor 300, corresponds to 8 
clock cycles, as appreciated by those skilled in the art. 
The PIC~400 then "examines "the ~s'tr"eam of ^f ixed" length ~ 
cells "on the fly" to identify the "classification" (that 
10 is, the protocol encapsulation sequence), capture matrix, 
and other control information for each packet (as 
described more fully in copending Application No. 
09/494,235 filed January 30, 2000 entitled "Device and . 
Method for Packet Inspection, " the disclosure of which is 
15 incorporated herein by reference) . More specifically, 
the preferred PIC 400 generates a control cell for each 
examined cell of a packet, and each control cell • 
represents the control information that has been 
determined thus far for the corresponding packet- Thus, 
20 the PIC 4 00 outputs both the stream of fixed length cells 
that was produced before this stream was examined "on the 
fly" therein, as well as corresponding control cells. As 
shown in Fig. 3, these control and data cells are then 
provided by the PIC 400 to four preferably identical 
25 buffer access controllers ("BACs") 402-408. Each BAC 

stores a different quarter (i.e., 25%) of the data cells 
received from the PIC 400 in its corresponding cell 
buffer ("CB"). 

Each control cell output by the PIC 400 also 
30 includes a protocol processing unit ("PPU") identifier 
which identifies a PPU associated with a particular BAC 
for processing that control cell. Note that each PPU, in 
this preferred embodiment, preferably comprises two 
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general purpose central processing units ("CPUs"), as 
shown in Fig. 3. Alternatively, a PPU could comprise one 
or more network processors, digital signal processors, or 
any prograininabl"e~processors^ — The— BACs— 402-408~each 

5 examine the PPU identifiers contained in the control 

cells delivered thereto over a bus by the PIC 400. When 
a BAG determines that the PPU identifier . in a particular 
control cell is identifying the PPU associated with thaf 
BAG, the BAG will forward the control cell to its 

10 associated PPU for processing, as described more fully 
below- Thus, while every BAG 402-408 in this preferred 
embodiment stores a quarter of every data cell in their 
associated cell buffers, each control cell output by the 
PIC 400 is acted on by only one BAG and its associated 

15 PPU. As a result of being so processed, the size of the 
control cell is much smaller than the typical size of a 
packet. This can significantly increase the utilization 
of the processor by reducing the I/O bandwidth which is 
the typical limiting factor in processor use. In this 

20 preferred embodiment, all control cells corresponding to 
a specific packet (and, more generally, to a specific 
user) are processed by the same BAG PPU on the line card 
380. 

Note that the PPU assigned by the PIG 400 for any 
25 given packet is performed according to configuration and 
control information received by the PIG 400 from a master 
PPU {"MPPU") 410, and can be changed by the MPPU 410 over 
time as necessary for PPU load balancing on the line card 
380. 

30 The PIG 400 also keeps track of the available memory 

addresses in the cell buffers associated with the BACs 
using a free buffer ("FB") list 412, and also keeps track 
of where each data cell is stored in the^ cell buffers 
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with respect to other cells of the same packet using a 
link list 414. 

When a control cell is processed within a particular 

BAC PPU, tfie"PPD produces a riewn:^Un'trol'~ce-l^^ 

5 provided to a packet manager ("PM") 420 which is in 
communication with the PIC 400 and the BACs 402-408. 
Included in this control cell provided to the PM 420 is a 
dequeue pointer which designates the location of the 
first cell of a packet that is to be dequeued and sent to 

10 the PM 420 along with the second and subsequent cells of 
that packet (if applicable) . The packet manager 420 then 
forwards this dequeue pointer back to the PIC 400, which, 
in turn, provides instructions to the BACs 402-408 to 
dequeue each quarter cell of the designed packet in 

15 sequence using the information previously stored by the 
PIC 400 in the link list 414, Thus, the designed packet, 
is reassembled as it is dequeued and delivered to the 
packet manager 420. 

At this point in the processing, the packet manager 

20 420 stores the cells of the reassembled packet in its own 
cell buffer 422 (using a free buffer list 424 and link 
list 426) . The packet manager 420 processes the control 
information it received for that packet from one of the 
BAC PPUs and then formats the packet according to this 

25 control information by modifying or augmenting the packet 
header as the cells of the packet are dequeued from the 
cell buffer 422. This process and additional details of 
the preferred packet manager 420 are described more fully 
in copending Application No. 09/494,236 filed January 30, 

30 2000 entitled "Device and Method for Packet Formatting," 
the disclosure of which is incorporated herein by 
reference. The packet manager 420 also appends a header 
to each of the 64 byte cells that constitute the 
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reassembled and formatted packet^ and these headers will 
be used by the switch fabric for routing the cells 
therethrough. The packet manager 420 then forwards the 
cell"s~of ~the~packet— in— sequence to -a-ODASL— 4-30 — whieh— is — 

5 provided for managing cell traffic into and out of the 
switch fabric for the line card 380. Typically, the 
UDASL 430 then forwards the packet cells into the switch 
fabric for delivery to an IPE~ card" that "Vill perfofm'mid- 
network processing functions for the packet in question. 

10 This IPE card is preferably designated by the BAC PPD . 
that prepared and forwarded control information to the 
packet manager 420. 

Also shown in Fig. 3 is a 9-port Ethernet switch 450 
which provides for interprocessor communications between 

15 the eight FPUs on the line card 380 (i.e., 4 FPUs on the 
ingress side and 4 FPUs on the egress side) and the MPPU. 
410 for purposes of load balancing, hardware monitoring 
and bandwidth distribution, and for sharing user and 
configuration information. The bandwidth distribution 

20 process and the preferred hardware are described more 
fully in copending Application No. 09/515,028 filed 
February 29, 2000 entitled "Method and Device for 
Distributing Bandwidth," the disclosure of which is 
incorporated herein by reference. 

25 Fig- 4 illustrates an exemplary IPE card 500 used in 

the. preferred mid-network processor 300 of the present 
invention. The hardware layout of the IPE card 500 is 
similar to the hardware layout on the ingress side (and 
the egress side) of the line card 380 shown in Fig. 3. 

30 That is, the IPE card 500 is also provided with a UDASL 

501 that delivers a typically interleaved cell stream 
received from the switch fabric to a PIC 502. The PIC 

502 is in communication with four BACs 504-510 that are 
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in communication with a PM 512. Thus, the primary 
difference between the preferred IPE card 500 and either 
side of the preferred line card 380 is the processing 
that is performed therein, even thougfi^tfiis prc5ces'sing~is^ 
5 performed with similar hardware. It should thus be 
apparent that the present invention provides, amongst 

other_ things^^ an_ inventive hardware module that can be 

programmed to perfonn requisite processing either on the 
ingress side or the egress side of a line card, or on an 

10 IPE card- This contributes to the configurability and 
scalability of the preferred mid-network processor 300, 
which can be reconfigured as necessary (both through 
programming and/or by adding additional lines cards 
and/or IPE cards) to accommodate additional users and/or 

15 to provide additional processing power. 

Much like the PIC 400 resident on the ingress side 
of the preferred line card 380, the PIC 502 provided on 
the preferred IPE card 500 is used to inspect the stream 
of fixed length cells provided thereto by the switch 

20 fabric "on the fly" to ascertain control information for 
each packet to be processed on the IPE card. In most 
cases, this control information was added to the packet 
by the PM 420 on the ingress side of the line card that 
forwarded the packet to this particular IPE card- The 

25 PIC 502 outputs the stream of data cells to the four BACs 
504-510, each of which is configured to store a different 
quarter of each data cell in its corresponding cell 
buffer (note that each BAG on the preferred IPE card 500 
has two FPUs associated therewith, whereas only one PPU 

30 is associated with each BAC on the preferred line card 

380) • The PIC 502 also outputs control cells to the BACs 
504-510, where each control cell contains a PPU 
identifier that designates one of the two PPUs associated 
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with a particular BAG for processing that control cell on 
the IPE card to perform mid-network processing functions 
for the corresponding packet. In this preferred 
emSodimenir; all'^colTtfol^c'eirs^cT?^^ 
5 packet (and, more generally, to a specific user) are 
processed by the same BAG PPU on the IPE card 500.. 

For any given packet, the PPU that processed control 
information for that paclcet on the ingress side of the 
line card is also responsible for determining to which 

10 IPE card and, more specifically, to which PPU on a 
particular IPE card, the packet should be sent for 
further processing. 

After a BAG PPU on the IPE card processes the 
control information for a particular packet, the PPU 

15 sends a control cell back to the PM 512, which then 

cooperates with the PIC 502 to dequeue the quarter cells 
of that packet in sequence from the cell buffers 
associated with the BACs 504-510. Upon receiving the 
constituent cells of a reassembled packet and storing 

20 these cells in its own cell buffer 514 (using a link list 
516 and a free buffer list 518), the PM 512 processes the 
control cell received from the BAG PPU to format the 
reassembled packet according to its destination interface 
before forwarding the reassembled formatted packet back 

25 into the switch fabric for delivery to its destination 
line card (or another IPE card, in the case where 
additional processing of the packet is required) . 

Also shown in Fig. 4 is a 9-port Ethernet switch 550 
which, like the Ethernet switch provided on the preferred 

30 line card 380, provides for interprpcessor communications 
between the eight PPUs and an MPPU 530 on the IPE card 
500 for purposes of load balancing, hardware monitoring 
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and bandwidth distribution, and for sharing user and 
configuration information. 

Referring again to Fig. 3, it can be seen that the 

egress side Qf the exemplary Tine card~380~i"s~a±so 

5 provided with a PIC 600, four BACs 602-608, and a PM 610. 
Upon receiving a possibly interleaved stream of fixed 
length cells from the switch fabric via the UDASL 430, 
the PIC 600 examines this cell stream "on the fly" to" 
ascertain control information (including control 

10 information that may have been added to the packet header 
by the PM 512 on an exemplary IPE card 500) . The PIC 600 
then forwards the data cells to the BACs 602-608 for 
storage in their corresponding cell buffers, and forwards 
corresponding control cells for each packet to one of the 

15 BAG PPUs (typically assigned by an IPE card BAG PPU that 
previously processed control information for the same 
packet) for further processing. The assigned BAC PPU 
then performs additional packet processing, primarily for 
traffic shaping, PHY card scheduling and bandwidth 

20 distribution on that PHY card. This process and the 

preferred hardware are described more fully in copending 
Application No. 09/511,059 filed February 23, 2000 
entitled Method and Device for Data Traffic Shaping," 
the disclosure of which is incorporated herein by 

25 reference. Upon processing the control information 
received from the PIC 600, this BAC PPU produces and 
forwards a control cell- to the packet manager 610, which, 
in turn, dequeues the quarter cells of the corresponding 
packet in sequence from the cell buffers associated with 

30 the BACs 602-608 in cooperation with the PIC 600. The PM 
610 then stores the constituent cells of the reassembled 
packet in its own cell buffer 612 (using a link list 614 
and a free buffer list 616) , and formats the packet for 
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its intended destination before forwarding, the 
reassembled formatted packet to the PHY card associated 
with this line card for outputting the packet froin the 



mid-network processor 300. 



5 A description of one preferred implementation of the 

broadband mid-network server described above will now* be 

provided, wherein the following terms have the following 

meanings: " 
Cardid: An 8 bit number that uniquely identifies an IPE 
10 or Line Card in the system. 

Flowld: A 10 bit number whose lower (least signif i<^ant) 
8 bits contain a Cardid, and whose upper (most 
significant) 2 bits identify the priority (class) of the 
traffic sent through the switch fabric to this card using 
15 this Flowld. (In the switch fabric, this field is 12 

bits, but our implementation only uses the least 
significant 10 bits.) 

Oser: A datalink (layer 2) interface. Examples include 
ATM virtual circuits, PPP sessions (over SONET, Ethernet, 

20 or ATM) , and MPLS label switched paths . 

Userld: A 32-bit value that can be used as a system-wide 
pointer to user configuration and state information. 
Since multiple cards (one or more IPEs and one Line Card) 
can store information about a user, it is possible to 

25 have multiple Userlds that refer to a single user. The 

upper (most significant) 8 bits of the value represent 
the Cardid of the card which contains the user 
information being identified. The next 4 bits represent 
the PPDID of the PPU on the card where the information is 

30 stored, and the lower (least significant) 20 bits 

represent the CID assigned by that card to the user. The 
CID is used as an index into the PPD's table of user 
information. 

LCUserld: A Userld in which the Cardid identifies a Line 
35 Card. 
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Primary Userld: A Dserld in which the Cardid and PPUID 
identify the PPU on an IPE with has the primary 
responsibility for managing a user. 

S&^i:on'dary-UserTd:— A-Userld— i-n-whieh-the-Gardld-and-P.puiD- 
5 identify an IPE PPD other than the one identified by the 

Primary Dserld 

Small User: A user whose ingress packet stream is 

- processed entirely- by-a single -IPE -PPD Small, users do_^ 

not have Secondary Userlds. 
10 Large User: A user whose configured bandwidth is too 

high for his ingress packet stream to be processed by a 
single IPE PPU. All large users have one or more 
Secondary Dserlds. 

Logical Link: A group of users of the same type (i.e.: a 
15 group of ATM Virtual Circuits) . If the Logical Link is a 

group of PPPoE sessions over ATM, the Logical Link must 
be an ATM Virtual Circuit. 

CSIX Header: The header of a CSIX (i.e.. Common Switch 
Interface) cell. The CSIX Header is separate from the 64 
20 byte cell payload. 

Cell Header: The first two bytes of the 64 byte payload 
of a CSIX cell. 

PIE Header: The 6 bytes immediately following the 
Cell Header of the first cell of a packet. 
25 Overview : 

In this particular implementation, the server system 
preferably comprises one or more rack mountable system 
units (i.e., shelves). The system also contains at least 
one line card, exactly as many PHY cards as line cards, 
30 and at least as many IPE cards as line cards. Also, each 
shelf of the system contains preferably three switch 
fabric cards and two flash disk cards. Each line card is 
uniquely associated with a particular PHY card. However, 
there is no particular association between line cards and 
35 IPE cards. 



wo 01/67694 



PCTA3S01/O1O03 



22 

Each IPE card can be thought of as an independent router, 
with one or more IP addresses associated with it. Each Layer 
2 (datalink) interface (referred to as a "user") provided by a 

-l4^ne-card-is-associated-with_exa.ctly_jDne IPE card (more 

5 specifically, exactly one PPD on one IPE card) . Different 

users from the same line card can be associated with different 
FPUs on different IPE cards, and a particular PPU can have 

.users from multipJLe_line_card^. _ _ 

Since it is possible for multiple Layer 2 protocols to be 
10 encapsulated within each other (for example, 

PPP/Ethernet/ATM) , there is an exception to the ^^one user, one 
PPU" rule. In this case, the inner-most levels of 
encapsulation, each of which being layer 2 interfaces (users) 
in their own right, can be associated with different PPUs 
15 within an IPE card, or even PPUs on different IPE cards, thus 
causing traffic from the outer levels of encapsulation to be 
split among multiple PPUs or IPE cards. It is also possible 
for outer layers to be encapsulated layer 3 traffic as well as 
layer 2 traffic (for example, an Ethernet/ATM virtual circuit 
20 can carry IP as well as PPPoE packets) . In this case, all the 
layer 3 traffic will be associated with a single PPU (a user) , 
but the encapsulated layer 2 datalinks (users) can each be 
associated with a different IPE card. 

The set of all users on the system is preferably 
25 distributed as evenly as possible across all the IPE cards in 
the system. Within an IPE, the MPPU stores the per-user 
information for the users assigned to that IPE and distributes 
those users across its PPUs . Each PPU stores a copy of the 
per-user information assigned to it. Thus each user is 
30 associated with one and only one IPE card and one and only one 
PPU on that IPE. This PPU' s copy of the user's configuration 
and state information can be uniquely identified on a system- 
wide basis by the Primary Userld. 

The architecture of this preferred in^lementation is 
35 based on line cards, PHY cards, a switching fabric, internet 
processing engines (IPE) and flash memory modules, as was 
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described generally above. The line cards terminate the link 
protocol and distribute the received packets based on user, 
tunnel or logical link information to a particular IPE through 
the-swi-tehi-ng-fabric — The-.procedure of forwardin g a pac ket to 
5 a particular IPE and PPD will be denoted as "routed , 
distribution." A midplane is also used to connect the 
different cards. The preferred line card and the preferred 
„ —jpE- card -were_described_abov^ with reference to Figs. 3 and 4. 

The system is comprised of a set of hardware components, 
10 as described, which can be used to configure a system for a 
wide variety of applications as well as throughput 
requirements cost effectively. The preferred switch fabric 
and scheduler support cell switching at OC-192 speeds, and the 
switch fabric is both fully redundant and highly scalable. 
15 The preferred IPE cards have the following attributes: high 
performance protocol processing engine; manages users, tunnels 
and secure segment groups; supports policing and traffic 
shaping; implements highly sophisticated QoS with additional 
support for differentiated seirvices; supports distributed 
20 bandwidth management processing; and supports distributed 

logical link management, able to do NAT, packet filtering and 
firewalls . 

The preferred line cards have the following attributes: 
packet lookup processing; protocol identification; scheduling; 

25 supports distributed bandwidth management processing; 

multi-I/F support (ATM, GE, POS) ; and AAL-5 Processing (CRC 
check and generation) . 

The preferred PHY cards have the following attributes: 
line termination for rates up to OC 192c; ATM - Layer 

30 Processing; ATM - SONET Mapping; POS - SONET Mapping 

(including FEC checksum computation) ; GE - MAC and PHY 
Processing; and support the following line cards: ATM: 4x OC- 
48, 8x OC-12, 16x OC-3; POS: lxOC192, 4x OC-48, 16x OC-12; and 
GE: 8/lOx GE. Additionally, the overall system preferably has 

35 the following attributes: high availability; 1+1 switch 
fabric and scheduler redundancy; 1+1 control system unit 
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redundancy; all field replaceable units are hot-swappable; N+1 
AC power supply redundancy; and N+1 fan redundancy. 

One purpose of routed distribution is to forward a packet 

^to-a-paJ^tiGular-PEU-wlthin^an^IPE.. The ke y be n efits o f thi s 

5 approach are: incremental provisioning of compute power per 
packet; allows load distribution based on the packet 
computation needs for a particular user or tunnel; user and 
- - tunnel- configuration information can be juaintained_by one 
single processor thus minimizing the inter-process 
10 communication needs; and allowing, the portability of single 
processor application S/W onto the system. 

Fig. 5 illustrates the distribution of packets to a 
particular IPE. A packet is received from a line card. The 
line card examines the packet and forwards the packet based on 
15 the IP source or destination address, the user session ID, or 
the tunnel ID- The IPE receives the packets and hands it over 
to the PPO specified by the line card. 

The line cards and the IPE host the flexible protocol- 
processing platform. This platform is comprised of a data path 
20 processing engine and the already mentioned protocol- 
processing unit. The separation of data path processing from 
protocol processing leads to the separation of memory and 
compute intensive applications from the flexible protocol 
processing recpairements . A clearly defined interface in the 
25 form of dual-port memory modules and data structures 

containing protocol specific information allows the deployment 
of general -purpose CPU modules for supporting the ever 
changing requirements of packet forwarding based on multi- 
layer protocol layers. 
30 The protocol-processing platform can be configured for 

multiple purposes and environments. That is^ it supports a 
variable number of general purpose CPUs which are used in the 
context of this architecture as Protocol Processing Units 
(PPU) . One of these CPUs is denoted as the Master Protocol 
35 Processing Unit (MPPU) . 
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The data path processing unit extracts, in the packet 
inspector, all necessary information from the received packets 
or cells and passes this . information on to a selected PPO via 
^one~o f — the-bu-f -f er-acce s s-cont r o lie r—de vice 5_. Th e cells 

5 themselves are stored in the cell buffer and linked together 
as linked lists of cells, which form a packet. Once a PPO has 
selected a packet for transmission^ it passes the pointer to 

the packet-and the necessary ^formatting and routing 

information to the data path processing unit. This enables 

10 the formatting and the segmenting of the packet. The packet 
is then forwarded either as a whole or segmented based on the 
configured interface. 

Each PPD is associated with one dual-ported memory, where 
one port is controlled by the data-path processing unit and 

15 the other by the corresponding PPD. Each dual-ported memory 
contains two ring buffers, where one ring buffer is used to 
forward protocol specific information from the data path to 
the PPO and the other is used for the other direction. The 
ring buffer for passing on protocol specific information to 

20 the PPD is called the receive buffer. The other buffer is> 

called the send buffer. Two pointers are maintained for each 
ring buffer. The write pointer for the receive buffer is 
maintained by the data path processing unit while the read 
pointer is set by the PPD. The send buffer's write pointer is 

25 controlled by the PPD and the read buffer by the data path 
processing unit. 
The PHY Card : 

The PHY card terminates the incoming transmission line. 
It also performs clock recovery and clock synthesis. Optical 

30 signals are converted into a parallel electrical signal which 
is then an input to a physical framer device which maps the 
incoming bit stream into the transmitted physical frame - 
Finally the physical layer of the corresponding link protocol 
processes the physical frames. In addition, link layer 

35 protocol processing is performed in order to provide a common 
packet interface to the line card. On the transmission side. 
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the packets or cells are mapped into physical frames. These 
frames are then encoded into the corresponding physical layer 
format and sent over the optical fiber to the receiving peer. 
— -T-he-physical-layer— format„is_pref erabl y either SONET or 
5 Gigabit Ethernet. The link layer format is preferably GE, ATM 
or PPP for POS. 
The Line Card : 

— ^ - The_line_ card__ performs packet^ fqrwarding_ for jthe egress 
and ingress path- Full duplex 10 Gbit/s throughput is 

10 provided. The line card interfaces to the PHY cards and the 
switch fabric card. The Line Card is preferably configured 
for either POS-PHY or UTOPIA III interface to the PHY card. 
The Line Card preferably hosts two Protocol Internet Engine 
(PIE) chip sets. On the ingress side, one PIE chip set 

15 supports four protocol-processing units (PPU) and one MPPD. 

The Four PPDs perform routed distribution to the various IPEs 
in the system. They also provide traffic shaping and 
scheduling of flows to the switching fabric. The remaining 
MPPD is used for overall control and supports the distributed 

20 bandwidth allocation protocol of the switching fabric. 

The Packet Inspector (PI) first examines incoming cells 
or packets and the protocol information is extracted based on 
matched patterns in the data flow. This information is then 
made available to the PPU which is responsible for processing 

25 the incoming packet. Cells or packets from a PHY card are 

processed by a particular PPD based on a chosen configuration. 
This configuration depends upon the configuration of the PHY 
card itself and upon the protocol supported by the PHY card. 
The other PIE chip set, processing the egress flow, is 

30 . preferably responsible for cell, assembly from the switch 
fabric and packet scheduling for multiple physical ports. 
Additional support for AAL5 processing is provided for ATM 
flows. The MPPU from the ingress path is shared for 
configuration, maintenance and cell extraction of the egress 

35 flow. 



wo 01/67694 PCT/USOl/01003 

27 

The coiDiaunication channel provides signaling and 
connection setup control for the ATM PHY card. The PHY card 
informs the Line Card about the physical layer status and 



reports alarm and error~conditions— 



5 The ingress packet processing preferably involves: 

Packet Assembly for ATM traffic (AAL5 processing) ; Protocol 
Identification (Packet Data Inspection); Routed Distribution; 
"Scheduling "of~traffic flows through switching -fabric;. Buffer 
management for ingress cell buffers; and cell scheduling for 

10 the switch fabric. 

The egress packet processing preferably involves: 
Traffic Shaping; Packet Assembly for switch fabric flow; MPHY 
Buffering; Cell Scheduling for ATM with multiple physical 
interfaces with AAL5 processing (CPCS, SAR) ; and Packet 

15 Scheduling for POS with multiple physical interfaces. 

The Internet Processing Engine (IPE) provides the 
functionality for protocol processing, user management r tunnel 
management and secure segmentation. It receives the packets 
from the switching, enforces the service level agreements 

20 (SLA's), performs packet classification, filtering and 

forwarding, and finally schedules the packet for transmission 
to the requested interface. 

The PI is part of the Packet Internet Engine (PIE) chip 
set, which consists of the Packet Inspector^ the Buffer Access 

25 Controller, and the Packet Manager. Together with the sixteen 
PPUs and the MPPU, the PIE chip set provides a powerful 
Protocol Processing unit. The PIE chip extracts informative 
protocol information and forwards it to the PPUs and the MPPO 
based on the routed distribution decision made in the Line ' 

30 Cards. The chosen PPU processes this information and performs 
all necessary packet processing. This includes, besides 
forwarding and filtering, policing, and packet formatting. 
The MPPD controls the IPE and is negotiating with the units in 
the system the bandwidth allocation of the switch fabric. It 

35 also provides bandwidth management for the configured logical 
links. The MPPD manages its connections by assigning users and 
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tunnels to individual PPOs for forwarding processing from the 
Line Card to a particular IPE» Once a connection between the 
MPPO and Line Card is set up, all packets belonging to such a 
c<5!m"ection— are-forwarded Hf-rom--the—Line^Card_^ 
5 A PPU is chosen based on the already assigned 

connections, their bandwidth and the bandwidth and QoS 
required for the new connection. Connectionless traffic 
- (Internet-to Internet ) is mapped onto ^an .internal _connect_ipn_. 
If more bandwidth is needed than one PPU can manage, the 

10 packets will be distributed over multiple PPDs. 

The functionality of the IPEs include: User Managements- 
Tunnel- Management; Logical Link Management; Support for Secure 
Segmentation; Policing; QoS Control with Diff Service Support; 
Buffer Management; IPv4, IPv6 Forwarding; Packet 

15 Classification; Packet Filtering with support for user 

Filters; Celox Management Database Support; Packet Formatting, 
and NAT. 

The Protocol Internet Engine Chip Set (PIE) : 

The Protocol Internet Engine (PIE) provides the data path 
20 processing capabilities for the server system at OC-192c 
rates. The PIE chip set comprises three chips. These chips 
result in a very high performance packet processing system 
together with an' interface contfroller and multiple general 
purpose CPUs. 

25 Each cell is preferably transferred into the buffer 

through four buffer access controllers ("BAGS'*) in order to 
increase . the bandwidth to the PPDs and to increase the 
bandwidth to the external cell buffers. Different portions of 
the same cell are written to the cell buffers attached to the 

30 different BACs. However, the captured portion of the data is 
sent to just one of the PPUs. 

The preferred BAC unit is shown in Fig. 8. The RSU 
receives incoming data, reformats the data ta an internal 
format, performs a parity check for incoming data, and also 

35 performs synchronization control- The preferred format of a 

cell received by the BAG from the packet inspector is shown in 
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Fig- 9. Referring again to Fig. 8, the Cell Filter unit 
extracts control information from the cell and sends the cell 
data to the BAU along with the indication of which portion of 

"tHe~ceTl—has~to-be— stored -in-thi-s-ce 1 1-buf f er ^The_CFJDL.aJLso__ 

5 sends the cell data stream to the PTU which translates, the 
PPUID to the appropriate PPU and thence to the CCQ where, 
based on the PPUID and the capture matrix, the control cell is 
" extracted -from the- data cell CCU. and-Stored in__the CBD. The _ 
CMD then transmits the control cell to the appropriate PPDs 
10 through a dual port RAM interface. 

When the packet has to be dequeued, the control cell 
corresponding to the packet is sent by the PPU which processed 
that user to the PM along with the dequeue pointer. This is 
received by the BEU of the PM, as shown in Figure 8. 
15 The control cell data stream (shown as the narrow arrow 

in Fig. 8) then goes to the ICU where it is stored while the 
DSD does deficit round robin scheduling of the data packets 
corresponding to the control packets in order to distribute 
bandwidth equitably to the BACs for sending out packets. In 
20 addition, the dequeue pointer corresponding to the packet to 
be dequeued is sent to the PID from where it is transmitted to 
the PI where it is received at the PIU and passed on to the 
BMU. In the EMU, the dequeue pointers are stored in a FIFO 
while the previous packets are being dequeued. The dequeue 
25 pointer information is passed onto the BACs and the BAD in the 
BACs. dequeues the packet and passes it through the PMU to the 
packet manager. A packet is dequeued by decjueuing all the 
cells comprising the packet which are held in the form of a 
linked list. Data packets from the data packet stream (shown 
30 as the thick arrow in Fig. 8) undergo AAL5 processing (should 
they need it) in the APU, and are stored in the IDD buffer. 
The FAD reformats packets into 64 bit slices and controls 
dequeuing from both the IDD and the DSD's DPRAM in accordance 
with the PFD- In order to ensure matching of the control 
35 packet with the data packet, a sequence number is used at the 
beginning of both the data and the control cells. Both the 
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control and data streams enter the PFO where they are 
formatted and sent to the TID to be sent to the phy cards or 
the switch fabric. 

" -The-P-I-E-chip-set-can-be_configured_fQr_muitiple_purpose^ 

5 and environments. That is^ it supports a variable number of 
general purpose CPUs which are used in the context with the 
PIE chip set as Protocol Processing Units (PPU) . One of these 

CPUs is reserved -for. maintenance and cpntrol purposes and is^ 

denoted as MPPU. 

10 The PIE chip set implements all necessary functions in 

order to hide all data. path processing from the actual 
protocol processing functionality- The PIE chip set extracts 
all necessary information from the received packets or cells 
and passes this information on to a selected PPO. The cells 

15 are then stored in the cell buffer and linked together as 

linked lists of cells, which form a packet. Once the PPU has 
selected a packet for transmission, it passes the pointer to 
the packet and the necessary formatting and routing 
information to the PIE chip set. This allows formatting and 

20 segmenting of the packet. The packet is then forwarded to the 
MPHY scheduler as a whole or segmented based on the configured 
interface . 

Each PIE chip set is differently configured. The PIE chip 
set on the IPE supports as many as 8 PPUs and 1 MPPU. 4 PPUs 
25 and 1 MPPU will support the PIE chip set on the ingress side 
of the Line Card, and an equal number on the egress side of 
the Line Card. 

The characteristics of the preferred PIE are as follows: 
Three Chip Chip-Set; Full Data-path processing in hardware; 

30 Support for distributed .protocol processing by general purpose 
CPU modules; Highly scalable compute power per packet (up to 
64 PPUs can be supported) ; Flexible interface support with 
MPHY scheduling; AAL-5 Processing; SAR Sublayer: Assembly and 
Segmentation for up to 256K connections; CPCS Sublayer: CRC 32 

35 generation and check, padding control, and length field 

control; Internal Packet Processing; Checksum computation and 
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check; Length field control; Padding control; Micro- 
programmable Packet Inspection Engine; Supports any layer 
packet inspection; Supports byte matched pattern processing; 
Supports-bit— inatehed-pattern-processing;__.Rej5UJ/bs are made 
5 available to protocol processing units; Supports extraction of 
any portion of packet for protocol processing; IPv4/IPv6 
Header Checksiom; Congestion Avoidance Support; EPD; PPD; 

Internal -Back-pressure control; .Linked Lis^ Con trrql; Supports 

up to 8 million 64 byte cells (initially a million); Links 
10 cells together to form a packet; Garbage Collection; Assembly 
aging control; Buffer Access; Parity generation and check for 
signal integrity; Cyclic access for data rates up to 12Gbit/s; 
PPD Access; Dual-Port access control for up to 8 dual-port 
RAMs each with 512/256 KByte memory;. Support for dual-port RAM 
15 data synchronization; Dual-ring buffer control for each dual- 
port RAM for data exchange; Threshold-based access control for 
writes to ring buffer; Support for up to 24 Gb/s throughput 
(bi-directional) ; Back-pressuring in case of buffer overflow; 
Cyclic Packet Scheduling; Packet Scheduling for cyclic access 
20 control with support for data rates up to 12Gb/s; Micro- 
programmable packet formatter; Supports insertion, removal and 
overwriting for any byte in a packet at OC-192 speeds; 
Supports IPv4/lPv6 Header Checksum generation; Support UDP/TCP 
checksum generation; Cell Scheduling, Buffering and Linked 
25 List Management; Supports cell buffering for up 512K cells; 
and supports scheduling for up to 1024 queues. 

Together with the PPOs, the preferred PIE supports: 
Packet Classification : 

Based on Layer 3,4,... Information (any layer); Packet 
30 Filtering; User programmable filters; Group filters; Firewall 

processing; Packet Forwarding; IPv4 Lookup Processing; IPv6 

Lookup Processing; Tiinnel Forwarding; Buffer Management; 

Dynamic Thresholding on a per user and assigned rate basis; 

Support for up to 8 million Cell Buffer (initially a million) ; 
35 Congestion avoidance with Early Packet Discard (EPD) , Partial 

Packet Discard (PPD), Selective Packet Discard; Policing; Per 
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User and Logical Link; Enforcing traffic contracts based on 
SLA; Traffic Shaping; Per User and Logical Link; Support for 
traffic contracts based on SLAs; Support for Real-time traffic 

(-low-delay— traf-f-iG)-;—QoS-Control;—Supported~for-differentiated 

5 services; Multiple priorities per user; Flow based queuing 
(not initially supported); Bandwidth Management; Distributed 
processing for allocation of bandwidth on switch-fabric links 
- including -MPHY- links;- Distributed, processing _f or. allocation of 
bandwidth for logical link management; User Management for up 

10 to 512K users; Tunnel Management for up to 128K users; L2TP; 
IPSec; Multi Protocol Processing; and Support for any 
protocol . 

Traffic Management : 

Traffic Management for an Internet access system is 

15 complex due to the involvement of various system interfaces. 
A system might be connected to users, the Internet backbone/ a 
Local Area Network with file and Web servers, and a 
Metropolitan Area Network (MAN) which gives access to local TV 
and media servers as shown in Fig. 11. Bach link has 

20 different link properties with respect to available bandwidth 
and Dollar per Megabyte. This means that a user's share of 
bandwidth on a particular link has to be based on the property 
of this link. A user might get more bandwidth share on the MAN 
link than on the backbone link due to the fact that more 

25 bandwidth at a cheaper price is available on the MAN link than 
on the backbone link- The same is true for bandwidth 
wholesaling of the preferred system to multiple ISPs who would 
like to resell bandwidth to their customers. The enabling 
technology for this model is Secure Segmentation. This model 

30 has also led to the introduction of logical link groups. A 

logical link group can be assigned to a secure segment based 
on the bandwidth needs of the considered secure segment for a 
particular link as shown in Fig. 12. This means that not only 
user allocation has to be considered but also logicsil link 

35. bandwidth needs to be included. Therefore, bandwidth is 

distributed based on traffic class, user, and logical link 
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group. This supports the wholesaling model and takes into 
account over-subscription requirements in order to support QoS 
including differentiated services. 

The_pr.e f er r.ed_3y.sj: ean .rjspr e se.nt s _a_Mghl y_di sir ij^u^^^ 

5 system. In such a system, resources have to be allocated based 
on the requirements of the traffic of each component. That 
means in general that each component has to take part in a 

distrvibujied j:omputation^m^ .9_^derJ:q^ allocate the 

resources. The traffic management requirements for bandwidth 

10 allocation within the preferred system will have to include 
bandwidth negotiation for the various flows through the 
switching fabric. One also has to consider the specific 
requirements to support the above-introduced concept of 
logical link groups- Since logical link groups are managed in 

15 a distributed manner, bandwidth information has to be 

exchanged between the entities managing one logical link as 
shown in Fig. 13. Buffer management and QoS Control is an 
integral part of the overall traffic management schemie 
implemented in the preferred server system. Due to the large 

20 buffer, the system has to maintain on various different places 
in the distributed system a sophisticated buffer management 
scheme which has to be implemented and supported by QoS 
control in order to support differentiated services and other 
traffic flow specific requirements 

25 Policing ~ Traffic Shaping : 

Policing and Traffic Shaping have closely related 
functionality. Policing ensures that the incoming stream does 
conform to the negotiated link parameters for a logical link 
group as well as the user of the incoming link. Traffic 

30 shaping enforces the link parameters for the outgoing traffic 
stream based on the outgoing user, the logical link group and 
the link itself. Fig. 14 is intended to illustrate the need 
for policing as well as traffic shaping. An incoming traffic 
stream is shaped (policed) in order to enforce the traffic 

35 contracts of a user for the considered link and logical link. 
Before the traffic is forwarded to another link, the traffic 
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contracts for this particular link have to be enforced. This 
traffic contract might be much different from the traffic 
contract of the incoming stream. Consider the case where a 

user requests information ove r the Internet backbone link, 

5 The bandwidth allocated on this link for this user might be 
500 Kbit/s. The logical link bandwidth for the corresponding 
secure segment might be set to 10 Mbit/s, If the user's 

access _lin_k to th^ system uses an ATM coiinecjtion with an 

assigned rate 1 Mbit/s and no policing is enforced, the user 

10 could use the full 1 Mbit/s. This is possible since the 
traffic shaped onto the user ATM link allows the user to 
transmit the higher rate. Therefore, it is necessary to police 
the incoming traffic and the other for shaping the traffic for 
a particular link. 

15 Fig. 15 shows the schematic implementation of the policer 

and traffic shaper in an IPE within the preferred server 
system. A received cell is assigned to a particular user data 
structure assigned to the incoming link for the considered 
user. As discussed earlier, the policing information can be 

20 directly obtained from the user who is sending a packet based 
on the connection identifier, the corresponding session ID, or 
the IP source address. However, if the packet on the incoming 
connection cannot be directly associated with a user or 
logical link group, then the user and/or logical link group 

25 for whom it is destined classifies the packet. Based on the 
obtained user and logical link information the incoming 
traffic stream is policed by queuing up the packets and 
enforcing the negotiated traffic contract. 

Once the packet conforms to the incoming link 

30 requirements, the packet is shaped based on the user 

parameters and logical parameters for the outgoing link. These 
parameters are obtained from the user connection itself if a 
session ID can be associated with it. If . the packet comes from 
a user and is forwarded across the Internet to a remote 

35 terminal, then the shaping parameters are obtained from the 
sending user for the corresponding link and the associated 
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.logical link group. For connectionless traffic, which cannot 
be directly associated with users, logical link group can be 
assigned based on the IP destination address and or source 

address. This allows managing tra ffic flows between networks. 

5 Switch Fabric Bandwidth Management and Scheduling : 

In order to meet the QoS requirements of individual 
traffic flows and to ensure that delay requirements of certain 

flOLWS can be met , sophisticated scheduling jaiust be conducted 

across the entire switch fabric. This scheduling takes into 

10 account the allocated user bandwidth, logical link share, 
buffer occupancy for output queues, available sxib-port 
bandwidth, priority of class of traffic, and expected delay. 
All this is accomplished while maintaining high throughput 
across the switch fabric. 

15 Attached hereto as Exhibit A are details of the 

manner in which the preferred server system is programmed 
so as to minimize inter-IPE card communications. 

There are various changes and modifications which 
may be made to the invention^ as apparent to those 

20 skilled in the art. However, such changes and 

modifications are suggested by the present disclosure, 
and the invention should therefore be limited only by the 
scope of the claims appended hereto, and their legal 
equivalents. 
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EXHmiTA 

Une Caids -^Ingress Side 

Line Caids do not pexfcHin any trafiic poficfng. Policing is peifonned, m distributed fashion* by the 

jJl IPE cards in Ae systemr-If during testingr^^ 

processor and VO bandwidth to perfoim policing, this function might be moved to the Line Cards in a 
future version of the software. AIso^ Une Cards do not petfoim any loutiDg table lodoipsL 

All IP padcets received, regardless of &en^ encapsuIaticMi, must have their destination IP address 
. cqituied and examined by a LC PPU. One opeiaticm diat nmst be petfonned is detemmiing if the 
• destination IP address is one oflhen* addresses of our systenL This can be done using 

~ table. A fufi QDR routing seardi is not necessaiy/since we are oni^ lo(&it]^ f^ 

result ofthelookiq) Of successful) is the Cardldof the IPEdiat the address belongs to^ Ifamatdiis 
found and the Cardid is equal to the Cardid o f the IPE diat the packet is about to be forwarded to, the 
packet must be forwarded with die Destination FPU bit set This is so ikaX when die padcet is 
received, die PI can select the packet to be captured in its entirety, (as long as it is iu>t part of a non- 
encrypted tunnel). 

Additioiialfy» if tiie packet is an IPsec padset that has been received fiom a large user^ and tiie 
destinatxon IP address is one of die addr^ses of the IPB to wfakh die pallet is 
to, die Usorld should be determined based on the IPsec Security Paraxn^ Index (SPI) radier than on 
die hash of die source and destination IP addresses in the IP header of the padcet These opoations 
win be discussed in greater detail in die sections diat follow. 
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JL1.2 ATM Une Card 
i.l^i Small Virtual Circuits 

The foUowing infonnation is sent to ftc IPE PPU along with the packet payload: 

• In the cm i/coJer; 

• DesdnationJFlowId: - i.^*.. 
S^t in Ac-Cm /Zcarfcr of every cel^of the pac^^^ 

should send it as well as the priority (class) of the packet 

• Initio Cdl Header: 

• SourceFlowId: 

Sent in flie Cell Header to allow the IPE to reassemble the packet This is siii5)ly the 
identification of the line card (m the least significant 8 bits) and die priority (class) in the 

. . most significant two bits. „The priority MUST be ttie saamc as spa:ificd m Ae 

Destination Flowld of this packet 

• Discard Bit 

Set by the LC PM to infonn the IPE tiiat an error (IP checksum, AAL5 C31C, mtemal 
parity) was detected in flie packet 

• EOPBit 

Set by the LC PM to indicate die last cell of flie pack^ 
• Jnib& PIE Header: 

• InltialPID: 

This 3 bit ficM tells die IPE the type of enc^[>subtionfliispai^ The choices are: 
IPC (for inter-piocessoT commumcations). IP, PPP, Bflieraet, ATM. «: MPLS 

• Initial Stage: ' 

Tliis 4 bit field can be used to give additional infonnation to die IPE about the 
encapsulaticm of diis packet It specifies which stage in die IPE PI will be die first to 
inspect die packet 

• DestinationPPU(lbit): ™ 

ITiis bit is set for IP padcets whose destination address is equal to one of Qie IP addresses 
of die IPE card diat packet is being sent to. 

• Destination PPUID: 

The PPU identifier of die IPE PPU diat die packet is being SCTt to. 

• IPEOD: , . , . 

An index into die connection table of die IPE PPU diat die packet is bcmg sent to. 

The PI uses die WWa and PHYm to calcukte die IX: Cro, which is used by die PI a^ 

die hardware connection table- The PI reads (amongst odicr diin©j) a LC PPUTO which selects 

LC PPU diat die control infoEniation for die packet should be sent to. 

The LC CSD is also used by die LC PPU as an index into a software connection table. Typically 
(diongh not always), diis connection table is used to determine die UserJd (which consists of a 
Desti^n Cardld. Destmation PPUm, and IPE OD) diat is sent to 

padcet As packets arc in^)ectcd by die LC, a determination of priority (one of four classes) is made 
^sed on die protocols found in die padcet Ahcmatively, die pritnity could be read firom die 
connection table. This priority is used to determine die two most significant bits 

Fiowld when the packet is forwarded to an IPE. 

The ATM ceU headers and die AAL5 trafler and paddmg are removed (by die PM) before forwardmg 
the padcet 

The foOowing is a list of all die different types of top level protocol encapsnlation diat canbe reodved 
by die ATM line card, alcmg widi an explanation of die fnocessing timt must take place cki die tine 
card for eadi type of protocol: 
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1.1:2.1.1 IP(RFC-1483) 

The BP packet is fcwwarded to the IPE. The IPE CID is dctcnnincd by reading the software 
connection fable. 

1.1^1.2 Ethernet (RFC.1483) 

Each ATM LC must have a standard globally unique Ethernet MAC address permanently assigned to 
it Each Ethemet/ATM VC should be conjBguiable as to whelber or not it is in "promiscuous** mode - 
that isrwhcther or not ft should discard unicast packets not sent to its MAGaddiess. 



AO Ediemet packets are forwarded to the IPE with dietr MAC headeis intact, except for PPPoE 
session packets (ethertype = 0x8864). For non-PPPoE session packets, the IPE CID is determined by 
reading the software connecticHi table, and the hdtial PID is set to indicate an Ethernet packet 

_ForJPPPpE_session packets, the PPPoE h^dar_is reimwe^ and die PPPoE Session_ip_(fiona the 
PPPoE header) is used to mdex mto a PFPoE ses^on table, from 

In this case die Initial PID is set to indicate a PPP packet Additionally, if the PPP/PPPoE protocol 
type is IP, the PPP header is also removed before the packet is forwarded to the IPE, and the Initial 
PID is set to indicate an IP packet 

1.1.2.13 PPP 

If flic PPP protocol type is IP, the PPP header is removed, and &e Initial PIDS is set to mdicate IP, 
odierwise, ihc PPP header is kqpt, and tfie initial PIDS is set to indicate PPP. The IPE CID is 
detennincd by reading tfie software connedicm table. 

1.1J2L1>I MPLS 

The top of stadc shim label (in the AAL5 PDU) is rephccd with the VPI/VCI of the virtual circuit 
The VPI/va can be deduced fiom the LC QD. The IPE CID is detennined by reading die software 
connecdon table. 

1 AJZJt Laige Virtual Circttits 

Widi die exception of die following dianges, large virtual circuits are handled in die same way as 
small virtual cixcoits. 

The PI DFU control r^isteis can be pnigranomiied (by the MPPU) w^ 

virtual circuits. For these circuits, if die packet contains an IP header, die PI DFUwinrq)Uu»ih^ 
PPUID read from the hardware connecdon table with a LC PPUID read fiom a hadi ts^Ie which is 
indexed by a hash of die sooroe and desdnadon TP addresses of die packet (calcoIatBd by die PI DFU). 

Any of die entries (drcuits) in die software VC cminecti(m table can be marked for dtstribntion across 
multq>leIPEPPUs. These are kriown as tos^tisen, and iieed riot be die same virtual drcui^ 
distributed by the DFU as plained above. For diese drcmts, if die packet contains an IP header, a 
new hash is calculated ova die source and destination IP addresses of die packet and used to select 
one of sev«al Userlds (Destination Cardid, Destination PPUID, and IPE CID) that are sent to die IPE 
in the PIE Header of d^ packet The most significant bits of die Destination Flowld are still used to 
select die priority (class) of die packet Howe!ver,inthecaseof an IPsec packet addressed to the IPE 
card diat die padoet will be fcnwarded to, die Userld is selected using a dififerent means, as described 
iQ die IPsec protocol processing section below. 



JLl^ POSUneCaM ^ ^ ^ 

The foUowmg infonnadim is sent to the IPE PPU along widi die padxt pay toad: 
• hkHieCSiX Header 
# Destination Flowld: 
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Sent in the CS!X Header of eveiy cell of the packet to identify iR^iere the switch fahric 
should send it as well as the priority (class) of tiie packet 

• hxthc Cell Header: 

• Source Flowld: 

Sent in the Cell Header to allow the IPE to leassemble the packet Tfiis is simply the 
identification of the line card (in the least significant 8 bits) and the priority (class) in the 
most significant two bits. The priority MUST be the same as is specified in the 
Destination Flowld of this packet 

• Discard Bit 

Set by the LC PM to inform the BPE that an eiror (IP checksum, internal parity) was 
„ detected in tfie packet* ~ ~ ~ 

• EOPBit 

Set by the LC PM to indicate ttie last cell of &e padcet 

• In PIE Header: 

• Initial PID: 

This 3 bit field tells the BPE the type of eocs^snlation Ifats packet has. The choices are: 
IPC, IP, PPP, or MPLS 

• Lutial Stage: 

This 4 bit field can be used to give additional information to die PE about me 
encapsulation of this packet It specifies whidi stage in the IPE PI will be the fiist to 
inspect die packet 

• Destination PPU (1 bit): 

Tliis bit is set for IP packets ^ose dcstinaticm address is equal to one of Ae addict 

of die IPE card that packet is beii^ sent to. 

• Destination PPUn): 

Hie PPU identifi^ of the IPE FPU that flie padcet is bdng sent to. 

• IPEC3D: 

An index into die connection table of the IPE PPU flmt the paclcet is being sent to. 



Unless MPLS/PPP/SONET is being used, each PPP/SONET PHY con^rises a smgle user. When 
MPLS is in use, howevor, cadiMPLS Labd SwitdiedPafh ^) rejxesents an additional user. 

For POS Line Cards, die LC OD is sin^ily die PHYID. The PI DFU ocmtrol roisters can be 
programmcd(by dieMPPU)wididieLCC3D'sofi^to4PHYs. For tiiese PHYs, if die packet 
contains an IP header, die PI DFU wifl replace die LC PPUID read from die hardware connection 
table wifli a LC PPUID read fiom a hash table whidi is indexed by a hash of the source and 
dcstinationIPaddic«csofdiepadcet(cakulatedbyfliePIDFU). Thiscapability of dicPIDFUnmst 
be used for OC-192c and OC48c PHYs in order to distribute the load over multiple LC FPUs, For 
OC-12candsniallCTPHYs,diePIDFUneednotbeused. Instead, die PI uses die U: OD as index 
mto die hardware connection table. The PI reads (aniongst odier dimgs) a LC PPUID which selects 
the LC PPU diat die control information for the packet should be sait to. 

As padccts are inflected by die lA a drtennmation of priority (one of fw This 
priority is used to detennine die two most significant bits of die Destination Ftowld y^fbrn die pactet 
is forwarded to an IPE. 

The following is a hst of all die different types of top level protocol encapsulation diat can be received 
by the POS line card, along widi an explanation of die pipcessmg duit must take ^ 

for each type of piotocoL 

1.1^.1 PPP Control Protocol (LCP, PAP, CHAP, IPCP, MPLSCP, etc«^) 

This category includes not onty PPP Control Protocols, but also any Netwc^ Protocol odier dian IP 
or MPLS. 
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The LC PPU uses the LC OD (whkh is really just the PHYID) as an index into a software PHY table. 
This table provides the Primary Userld, which determines where the packet is sent as well as fte IPE 
CID that is sent in the PJE Header of the packet For large and small PPP/SONET users, all non-IP 
and non-MPLS packets are sent to the IPE PPU identified by the Primary Userld. No distribution is 
perfoiined for these packets. Also, die Lutial PID is set to indicate a PPP pa(^ 



1.1^.2 IP 

The LC PPU uses the LC CED (vMch is really just the PHYID) to index into and read from die 
software PHY table. From diis the LC determines whether this is a user or a Zarge itfer. For 
small users, iStt&Primioy Userld h^Tead&amfl^t This determines where the packet is 

sent as well as the IPB OD that is sent ia die PIE Header of die packet 

- — — — For /ar^e Msersrliowever, a"hash is calculated over Ihe source and destination - — — — 

packet and used to select eidier die Primary UserlD or one of several Secondary UserlDs. The 
selected 6!s^er//> delcnmnes where die packet is sent as we^ 
Header of die packet 

The PPP header is removed before die packet is forwarded to die IPE, and the Initial PID is set to 
indicate an IP padcet 

1.1-3.3 MPLS 

As fa the case widilP/PPP/SOl^, die LCCro only identifies die PHYTO Therefore, when die LC 
PI identifies an MPLS padcet, die top of stack label most be captured in order to identify die user. 
For each POS PHY, ^ LC PPU most maintam a table of MPLS LSPs. The LC OD selects whidi 
tkble, and the top of stai^labd is used to index into die table. Vor small usm^ihs Primary UserlD 
diatcorreq)onds to die LSP can diea be read die table. For large loeriv, however, a similar process to 
die one described above for IP is used A ha^ is calculated over die source and destination IP 
addresses of die padcet and used to select eidier die Primary UserlD or one of several Secondary 
VserlDs. The selected UserlD detennines where die packet is sent as well as die IPE CID diat is sent 
in die PIE /feoi/er of die packet 

The PPP header is removed before die packet is forwarded to die IPE, and die Initial PD> is set to 
indicate an MPLS paclcet 

Ethernet Ui^ Card 

The folb wing inf onmation is sent to die IPE PPU aloiig widi die padcet ^ 

• lai^CSIX Header: 

• Destmation Flowld: 

Sent in the CSIX Header of every ceQ of die padcet to identify where die switch fabric 
should send it as well as die priority (class) of the padcet 

• In die C^i/eader; 

• Source Flowld: 

Sent in die Cell Header to allow die IPE lo reassemble die packet This is simply the 
identification of die line card (in die least significant 8 bits) and die priority (class) in die 
most s^nificant two bits. Tlie priority MUST be die same as is spedfied in the 
Destination Flowld of diis padcet 
^ • DiscardBit 

Set by the LC PM to inform the IPE that an enor (IP diecksum, internal parity) was 
detected in the padcet 

• EOPBit: 

Set by die LC PM to indicate die last cell of the packet 

• In Hkt PIE Header: 

• Initial PID: 
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This 3 bit field tells ihe IPE the type of encapsulaticm this packet bas. The choices are: 
IPC, Ethernet, PPP, DP, or MPLS. • 

• hiitial Stage: 

This 4 bit field can be used to give additional iiifonDati<m to the IPE about flie 

encapsulation of this packet It specifies which stage in the IP£ PI will~be Ihe fir^ to 
inspect the packet 

• Destination PPU(1 bit): 

This bit is set for IP padcets whose destination address is equal to one of the IP addresses 
of the IPE card that padcet is being sent to. 

• Destination PPUID: 

IliePPUi<tentifierofdicIPEPPUflatftepa<±ctisbeiD^ 

• IPECDD: 

An index into the connection table of the IPE PPU that the padcet is being s^ to. 

Unless MPLS or PPPoE is being used over (he Ethernet, each PHY conq;)iises a smgle user. When 
MPLS or PPPoE is in use, however, eadi MPLS Label Switehed Path (LSP) or PPPoE session 
represents an additional user. 

For Ethemrt Lme Cards, the LC C3D is sssoply the PHYID, The PI DFU control registers can be 
programn^(by AeMPPU)widi AcLCaD»sofiv to4PHYs. For Aese PHYs, if flie packet 
contains an IP heato, flie PI DFU will rq)lace the LC PPUID read from the hardware connection 
table wifli a LC PPUID read horn a Imsh table which is indexed by a hash of the source and 
destination IP addresses of file padcet (cakokted by tiie PI DFU). This capabihty of 
be used for 10 Gigs^itEdiemet Cards in order to distribute Ac load ov^nn Fori 
Gigabit and smaller PHYs,&e PI DFU need not be used. Instead, tiie PI uses tiie LC CD) as index 
into the hanhvare connection table. The PI reads (amongst othCT things) a LC PPUID ^ch selects 
(he LC PPU that the control information for the padcet should be scat to. 

Eadi Ethernet PHY must have a globally unicpie E&met MAC address permanently assigned to it 
AB Ediemet padcets are forwarded to die IPE with tbetr MAC headers mtac^ and tiie wiOi Initial 
FIDS set to mdicate Ediemet, excqpt for MPLS and PPPoE session padcets (ediectype - 0x8864). 

As padcets are inspected by the LC, a determination of priority (one of four classes) is made. This 
priority is used to determine the two most significant bits of the Destination Flowld when die packet 
is forwarded to an IPR 

The following is a list of all die difGamt types of top level protocol ettciq[>sulation that can be recdved 
by die Ethernet line card, along widi an explanation of the processing diat moost taioe place on the line 
card for each type of protocol: 



The LC PPU uses die LC CID (which is really just die PHYID) to index mto and read from the 
software PHY table. From this the LC determines whedier diis is a smaU.user or a large user. For 
^naU users^ Ihe PrmaryUserld is siisoitad from This determines ^idiere die packet is 

sent as weU as the IPE CID diat is sent in the PIE Header of die packet 

For large users, however, a hash is calculated over the scarce and destination IP addresses of the 
packet and used to select eidier die Primary UserlD or one of several &com2(iry UserlDs. The 
selected UserJD determines where die packet is sent as well as the IPE CID diat is sent in die PIE 
/fead^ of the packet 

The packet is forwatded to the IPE with the Ethernet MAC header intact, and the Initial FID is set to 
indicate an Ediemet padcet 
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i i 4,2 PPPoE Session 

* ' For PPPoE Session packets, the Ethernet and PPPoE headers arc removed, and the PPPoE Session ID 
(from the PPPoE header) is used to index into a PPPoE session table, from which the Userld (IPE 
Cardid, IPE PPUID and IPE CID) can be retrieved. A unique PPPoE Session table can be maintamed 
fi)r,each.PHY,.apd.flie IX:-Cro^can.be-ttsed.to sele<»j^^^ to use, 



If tiie PPP protocol type is IP, the PPP header is also removed, and Ae Initial PipSis set to indicate 
IP. oAerwise, the PPP header is kq)t, and flie Initial PBDS is set to indicate PPP. 

^'^ "^-fte LCGff) only identifies the PHYID that the packet was received on. Therefore, when the LC PI ^ 
' ' identifies an N0»I5 packetrt& top of sta^ 

each Ethernet PHY, Ae LC PPU must maintain a table of MPLS LSPs. TTie LC CID selecte which 
table, and the top of stack label is used to mdex into Ac table. For smaU MPLS users, the Prmuuy 
UserJD that corresponds to the LSP can flicn be read the table. For large users, however, a similar 
process to the one described above for IP is used. A hash is calculated ova: the source and destin^on 
IP addresses of the packet and used to select either the Primary UserED or one of several Secondary 
User IDs. Hie selected UserlD deteraiines where Ac padcet is sent as well as tiie IPE CID Aat is sort 
in flie P/£ //coder of Ae packet 

Hie Ethernet header is removed before the packet is forwarded to the IPE, and die Initial PID is set to 
indicate an MPLS packet 

^AAA Other Ethernet Protocols (ARP, PPPoE Discoveiy, etc) , ^ « . 

Ibis category inchides all Ethernet protocols (ethertypes) other than IP, MPLS, and PPPoE Session, 

Tbe LC PPU uses the LC OD (which is reaUy just die PHYID) as an index into a software PHY table. 
This table provides &e Prmary Userld, which detennmes where &e padcet is sent as well as the IPE 
OD that is sent in die P/E^«Kfcr of the padcet For J^c and smofl Ethernet usert. ftese pw^ets 
are sent to the IPE PPU identified by die Primary Userld, No distribution is performed for these 
packets. Also, Ae Initial PID is set to indicate an Efliemet packet. 

1.2 line Cards -^ress Side 

Line cards perform all the traffic shying for die system. 

1.2.1 ATM Line Card 

The foUowing information is received from IPE PPU along wiA the packet payload: 

♦ In the CSJX Header: 

• Destination Flowld: 

Sent in the CSIX Header of every cell of the packet to id«itify where flie switdi febnc 
should send it as well as flie priority (class) of the packet 

• Intbt Cell Header: 

• Source Flowld: , . . u ^ 

Sent in the Cell Header to aUow die LC to reassemble the packet This is sanpty the 
identification of die IPE card (in die least significant 8 bits) and die priori^ (class) in the 
most significant two bits. The priori^ MUST be the same as is specified m Ae 
Destination Flowld of diis packet 

• DiscardBit • . v ^ - 

Set by the EPE PM to mfonn die LC that an cnor (mteroal panQr) was detected m the 

packet 

• EOPBit 

Set by die IPE PM to indicate die last cell of the packet 
• laihc PIE Header: 

• InitialPID: 
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This 3 bit field tells the LC the type of encapsulation this packet has. The choices arc: 
IPC (for inter-processor c<Mnmunications), IP» Ethernet, PPP, or MPLS 

Initial Stage: 
Alwa ys 0. 



• Destination PPU(1 bit): 

Always 0. 

• Destination PPUID: 

The PPU identifier of the LCPPU fliat Ac padcet is beang sent to. 

• LCOD: 

This is an index into the connection table of the LC. 

The Destination PPUID selects tfieLC PPU that wiU process tfie packet The LC OD is used by the 
LC PPU as an index into a software cmmection table. This connection table provides the shaping 
parameters, any additional encapsulation &at must be added by the LC, thePHYID, and fte ATM 
VPWa for tiie packet ' 

The priority (one of four classes) is based on tiie two most significant bits of die Source Flowld in the 
CteB Header. Tbt prkmty is used by die Traffic Sh^ and flie SdieMo: to deteimme when to 
forward die padoet to file PHY. 

The ATM cell headers and die AAL5 tndkr and padding are always added (by die VM) heiote 
forwarding die padcet to die PHY card. 

The following is a list of all die diSeient types of top levd protocol encapsulatiim diat can be received 
fiom an IPE by die ATM line card, along widi an esqplanatixA of the processmg diat must take place 
on die line card for eadi type of piotocob 

^JZml i«i IP 

Tb& desired cncqisulati<Hi for die pacto can be eidier IP/PPP/PPPoE^Edicmet/ATM, IP/PPP/ATM or 
IP/ATM. The PPU can dcteranncvvhidi it is firom die connection table. If die encapsulation should 
be IP/PPP/PPPoE/Ediemet/ATM, die connection table will provide the necessary information to add 
die missing headers. If die caici^isulation ^ould be P/PPP/ATM, a PPP header identifying die 
]»otocol as IP is added. Also, die entry In die connecticm table may spedfy that an LLC header 
should also be added. 

1.2.1.1.2 Ethernet ^ , 

The connection table may speidfy diat an LLC header ^ould be added to die begnmmg of die packet 
Odierwise die padcet is sent as is. 

1.2.1.1.3 PPP 

The desired cncq)Sulation iray be cidier PPP/PPPoE/Ediemet/ATM or PPP/ATM The PPU can 
detemune which it is fiom die connection table. If it is PPP/ATM, die packet is sent as is, odierwise, 
die conuection table wiD provide the necessary informatimi to add a PPPoE header and an Ediemet 
Header. 

1.2.1.1^4 MPLS 

As widi all other encapsulations, the proper VPI/VCI is obtained fiom the connection table. 

1.Z2 POSUneCanf ^ ^ 

The foflowii^ mformation is received fiom IPE PPU alimg widi the padcet payload: 

• hxlheCSIX Header: 

• Destination FlowI± 

Sent in the CSIX Header of every cell of die packet to identify where die switch fabric 
shoukl send it as well as die priority (class) of die padcet 
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• Inihc Cell Header: 

• • Source Flowld: 

Sent in tiie Cell Header to allow the LC to reassemble tbe packet This is simply the 
identificatioii of the IPE card {in fee least significant 8 bits) and the priority (class) in tiie 
most significant two bits. The priority MUST be the same as is specified in the 
Destination Flowld of Hus padcet 

• Discard Bit 

Set-by-tlic-IPE-PM-to inforni-the-LG-that-an-eiror-(inteinal 

packet 

• EOPBit 

Set by the IPE PM to iiklicate the last cell of die padcet 

• In tbe PIE Header: 

• Initial PID: 

Tins 3 bit^fleld teUs the die type of cn«apsuktion_to packet_has. ^T^^ f ^i^^i 
- _ ~irc {foriater-processorc<mmiunicati^ 

• Initial Stage: 

Always 0. 

• Destination PPU (1 bit): 

Always 0. 

• Destination PPX3ID: 

The PPU idoitifia- of die LC PPU diat die paf^et is being sent to. 

• LCCaD: 

This is an index into the aHmectimi table of die LC. 

The Destination PPUID selects die LC PPU diat will process die packet TTie LC CCD is used by die 
LC PPU as an index into a software connection table. This connection table provides die shaping 
paiaineteis> and die PHYID for die packet 

The foUowmg is a list of all the different types of top level protocol encapsulation diat can be received 
by the POS line card> along with an explanation of die processing that must take place on the line card 
for each type of protocol: 

1A2.1 PPP 

No additicMial piocessing is needed. 

^JLZSt IP 

A PPP header identifying the packet as an IP packet is added. 

1.2.2.3 MPLS 

A PPP header identifying die packet as a MPLS padcet is added. 



1^ Bhemet Line Card 

The following mfonnation is received from IP£ PPU along with die packet payk>ad: 

• hiltitCSIX Header: 

• Destination Flowld: 

Sent in the CSIX Header of every cxXl of the padcet to identify where die switch fabric 
should send it as wen aslhepriorily (class) of die packet 

• In the Cell Header: 

• Source Flowld: 

Scut in die Cell Header to allow die LC to reassemble die pad:et This is simply die 
identification of the IPE card (in die least significant 8 bits) and die priority (class) in the 
most significant two bits. The priority MUST be die same as is specified in die 
Destination Flowld of diis padcet 

• Discard Bit: 
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Set by the IPE PM to inform the LC that an error (internal paiity) was detected in the 
packet. 
EOP Bit 

Set by the IPE PM to indicate the last ceil of the packet 
m—hi iht PIE Headers 

• Initial PID: 

This 3 bit field teUs the LC the type of encapsulation this packet has. The dioices are: 
IPC (for inter-processor coxnmumcations), Ethernet, PPP^ IP, or MPLS 

• Initial Stage: 

Always 0. 

_ _D.estinationPPU.(lb4t): _ 

Always 0. 

• Destination PPUID: 

The PPU identifier of the LC PPU that flie packet is being sent to. 

• LCCID: 

This is an index into the connection table of the LC. 

The Destination PPUID selects the LC PPU that will process tfie padcet The LC CID is used by the 
LC PPU as an index into a software connection table. This connection table provides ^ discing 
paiameters, and the PHYID fiu^ &e packet 

The foDowiog is a list of all the differ^ types of top level protocol encapsalatiQn diat can be xecetved 
by the Ediemet line card, along wifii an explanation of the processing that must take place on flie Ime 
. card for each type of protocol: 

Ethernet 

No additional processing is needed. All IP/Edieniet aie sent using this type because the IPE, not file 
LC inq>letn^ ARP, and dieiefore adds the Etfiemet header to alllP padcels befoie sendh^ fbm to 
&eLC 

PPP 

The desired encapsulation is PPP/PPPoE/Ethemet The connection table provides Ae necessary 
infomiatioa to add a PPPoE header and an Ediemet header. 

IP 

The desired encapsulation is IP/PPP/PPPoE/Ethemet A PPP header hidicating an IP padcet is added 
The connection table dien provides die necessary information to add a PPPoE header and an Ethnnet 
header. 

1JL3>I MPLS 

The connection table {M^ovides the infonnation needed to add an Ediemet header (the destination 
MAC address is all that is recpiired fiom tfie connection table). 



13 IPE Card 

1^.1 JPE Ingress Protocots 

All packets received by an IPE card fit)m the Line Cards (or from other IPEs) will be of one of the 
following types. The Initial PID field in the PJE Header vnU identify whidi one of Ihese types each 
padcet coxr^ponds to. If diere are mote dian 8 such types, the Initial Stage field in the PIE Header 
can be used to select a different stage to begin haspection, each of which allows 8 additional protocob 
to be identified by the /inUsui/ PiZ> field. 

The IPE CID and PPUID in die PIE Header of die received packet combine widi die Flowld to give 
dieUserld. Only die least significant 18 ofdie 20 bits of dielPE CID are used. 
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1^1-1 IPC -xn. t. i^u 

These packets are used for inter-processor coiranunication within the systeio. The PI should be 
prpgrammed to capture these packets to a FPU (as specified in the Destination PPU field in the PIE 
Header) in their entirety. 

1*3 '1*2 IP 

^Thcse lockets consist of only an lP packet-That isr 

might inchide TCP, UDP, ICMP, etc). This category does not include IP packets received over 
MPLS or over Etisonet 

These packets can come from either a POS LC, an ATM LC, or an Ediemet LC. The possible 
encapsulations that could result in such a packet are: IP/ATM, IP/PPP/ATM, IP/PPF/SONET, 
, IP/PPPA;PPoE^thOTiet, aiidff^PP^PPoE^ 

The IPE CID uniquely identifies the PPPoE Session ID, or Ac ATM Virtual Circait diat Ae packet 
was received on as well as the PHY/LC fliat it was received on. In the case of IP/PPP/SONET, Hie 
IPE OD win identify only Ae PHY/IX that die packet was recei^ 
all IP/PPP/SONET packets received firwa a particQlar PHY/LC 



1^1.3 PPP 

This categoiy conasts of all PPP packets received whose PPP protocol type was not IP or MPLS. 
These packirts can come film a POS IX; an a™ LC, or an Ethernet LC For fliose PPP sessions tot 
will be tunneled udng L2TP» &e IPE must add a new PPP header to Oie IP/PPP and MPLS^PP 
padcets, smce for those inotocols, Ac PPP header will have bc«i removed by the Line Card. 

ITie possible enca^jsulatiwis that could result in such a padkct are: PPP/SONET, PPP/PPPoE/Eflicmet, 
PPP/ATM. and PPP/PPPoE/Ethemet/ATM 

TTie IPE CID muqnely identifies the PPPoE Session ID, or &e ATM Virtual CSrcoit fliat &e padcet 
was received on as wen as fl^PHYALC that it was received on. InftecaseofPPP/SONET,ftelPE 
CID vnll identify only the PHY/LC Aat die packet was recdved on, that is, it will be constant for all 
PPP/SOl^ET packets received fiom a particQUir PHY/LC. 

1.3.1^ Ethernet 

This category consists of aD Native Eftcmct or E&emet/ATM packets rccdved except for MPLS 
(e^ertype = Ox????) and PPPoE data padcets (etheitype = 0x8864). This category also mchides 
padcets whose destinati<ni MAC address is not equal to fbt MAC addiess of Ae PHY on whidi Ae 
packet was received (broadcast ai^ multicast packets, and unicast packets if in promiscuous mode). 

The possible encapsolatiaiis that could result in sadi a padret are: ARP/Efiianet, IP/E&csnet, PPPoE 
Discovcry/EthOTCt. ARP/EAemel/ATM, IP/Eflicmel/ATM. and PPPoE Discovoy/Ethcmet/ATM. 

For Ethcmet/ATM, the BPE CID miiqucly idaitifies the ATM Virtual Circuit fliat Ihe padcet was 
received on as wdl as die PHY/LC that it was received on. In tibe case of Native Ethernet, the IPE 
C3D wiD identify (mly d» PHY/LC that die packet was received on, Aat is, it will be constant for all 
padc^ received torn a particular PHY/LC 

13.1.5 MPLS 

Thig category omsists of packets which begin widi an MPLS label stack. Ihese .can come fix>m a 
POS LC, an ATM LC or an Ediemet LC. 

The possible encsqisulations diat could result in sudi a packet ate: MPLS/PPP/SONET, 
MPLS/Edicmet, and MPLS/ATM. In die case ofMPLS/ATM, the Line Can! wiUbaverqslaced^ 
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top of stack shim label with the real label because the real label was encoded as the ATM VPI/VQ in 
the packet received from the network. 

The following encapsulations are NOT supported: MPLS/PPP/P PPpE/Ethemet, MPLS/PPP/A TM, 
MPI^PP/PPPoE/Etbcmct/ATMrandMPl^/Ethcmet/A'^ ^ 

The IPE CID uniquely identifies die same as the top of stack incoming top of stack MPLS label, as 
well as the PHY/LC that it was received on. In the case of MPLS/ATM die top of stack label has a 
one to one correspondence widi die ATM Virtual Circuit that die packet was received on. 



IPE Ingress Protocol Decxjding 

The following table shows die first two layers of protocols diat must be identified by die PI on die IPE 
for each packet that passes fluough it 



IP 




E&emet 


IP 


AKP 


PPPoEDiscoveiy 


MPLS 


IP 


PPP 


Control Protocols 



AH IP packets received by die IPE, whcdier still encapsulated wdi Efliemet, widi MPLS, or widiout 
enc^>su]ation» will fall into one of two catpgcdes: diose for which die destination P address is equal 
to one of die addresses of flie IPE, and diose for which it isn't In die case of Ae lattCT, die packet 
must be forwarded or discarded by the PPU. Butfordiefoinici; itinustbcdctciniinedwhedier ornot 
die packet can be processed entirely by die PPU. ot wbedier it must be swit to the MPPU fw forflier 
processing. If it must be sent to die MPPU, it must be captured in its enlirc^- 

AH IP packets received, r^ardless of dieir encapsulation, must have dieir destination IP address 
c^ytured and examined AH routii^ table seaidies are prafonned by the IPE cards. If die destinati<m 
address is one of flic systm's IP addresses, but not one of die IPE card's addresses, die pac^ 
be forwarded widi Destination PPU bit set 



1.3^ L2TP Tunnels , ..^^ . 

Each L2TP tunnel is handled entirdy by a particular IPE card. Eadi session wiflnn die tunnel must lie 
handled entirely by a particular PPU. This requirement comes primarily from die need to support 
sequence numbers on the data sessions: 

RF02661: ''Each peer maintams separate sequence numbers for the control connection 
andeach individual da$a session wUhin a tunnei. " 

Iherefc^ lar^e pPP users camiot be tunneled. All L2TP control packets are forwarded to and 
processed by the MPPU of die IPE card. 

1 L2TP Access Concentrator (LAC) 

Any PPP user can be sdected for L21P tunneling by die IPE MPPU. If a user is selected for 
tunneling, then die PPU receiving PPP packets from diat user must encapsulate diosc packets, first 
widi an L2TP header, dien a UDP header, and finally an IP header. The IP header's destination 
address will be diat of die ouifigured LNS, and die source address will be one of die IP addresses of 
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IPE. The resulting IP packet can then be forwarded using the standard IP forwarding procedure to file 
appropriate Line Card for transmission. It should be evident that tunneled PPP users on different IPE 
caffds will be placed in separate tunnels even if being tunneled to the same destination LNS. 

DP packets received from the LNS wiU be sent by the receiving Line Card to the IPE PPU associated 
widi &e mgiess interfecc {user). This PPU may well be on a different IPE card than the one handling 
the tumieL Jlliis.is.easilyJetermincd ftom.the.destination IP-addre 
PPU receivii^ the packet from the Line Card must forward Ae packet to flie IPE card handling the 
tunnel In addition. Ac L2TP Session ID can be used to identifir which PPU on that IPE card should 
receive the packet (this PPUID must be sent in the PIE Header so diat the receiving PI will know 
which PPU should receive the packet). This is done by always oicoding the PPUID of the PPU 
handling a particular session in the most significant four bits of Ae L2TP Session ID. 

RFC'266I: ''Since UW sesshns are niamed by'ide^^ 

only. That is, the same session will be given differera Session Ids by each end of the 
session. Session ID in each message is thai of the intended recipient, not the sender. " 

The PPU to whidi the packet is sent to can in tarn can de-encapsulate the PPP packet and forward it 
to the PPP uso' i^ntified by Ac L2TP session ID. 



1.3JL2.2 L2TP Network Server (LNS) 

When functioning as an LNS, L2TP packets received from die LAC will be fOTwaided, eiflier by a 
line Qffd or anoflicr IPE, to flie IPE handling die tunnel This is because die destination IP address of 
fliepacketwiUbeequaltooneofdielPaddiessesofthelPEbandliiigAetunw WidiindiatIPE,die 
PPU diat should process die L2TP session is idoitified using die most significant four bits of die 
L2TP Session ID* The PPU will de-encapsulate die PPP padcet, dien process die PPP packet as if it 
was received fr<Kn a PPP user. From diis point on, die prDccsang is die same as f« a "reaT PPP 
user. 

In die odier direction, packets ixdiich, when dicir desdnaticm IP address is looked iq? in die routing 
table, yield a destmation PPP user diat is assodated wfli a LOT tunnel instead of with a Line Card, 
must be sent to die IPE PPU handlmg die FB? user. This is because of die sequence number 
requirement of L2TP mentioned above. received diis PPU, die packet must have a PPP header 
added, as is the case wrdi a normal PPP user. Then,insteadof forwartogdjcpackettoaLineCard, a 
L2TP header is added, foDowed by a UDP header and an IP header. The IP destination address is diat 
ofdieLACatdieodicrcndofflietonneL Tbe resulting IP padcet can dira be forwarded using the 
standard IP f(»warding procedure t^ the aiq^ropriatB Line Card for trans^ 



1^.2L3 IPSec Tunnels 



Eadh IPSec Security Association (SA) is handled entir^ by a particular IPE PPU. As defined m 
RFC-2401, a Security Association is a unidirectional, -simplcjT connection diat provides security 
services to die traffic carried by it 



1 ^ Jt3.1 Inbound IPsec processing 



• Plainpadcets 

Hveiy PPU most have a copy of die SPD for every user from which it receives packets. In other 
words, for every UserlD {Primary or Secondary) diat points to a particular IPE PPU, die PPU nrast 
have a pointer to an SPD. If a user's traffic is spUt znnaag multiple PPUs (i.e.: a large user), dien diey 
should have identical SPDs configmed for die user, and eadi will create its own set of Security 
Associations for its share of die iireKs traffic. Every pa<±^ received must be processed uang die 
SPD of the user die jpacket is received finm. 
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• Tunneled packets 

The SPI is tbe field in the IPSec header that, along witfi the destination IP address, identifies flic SA. 
Traffic from a small user will always be directed by receiving Line Card to a particular FPU. This 
FPU uses the SPI to identify the SA, and dius has access to the information it needs to dcc^sulate flie 
packet. For large users, however, the Line Card must detect IPsec packets whose IP destination 
address is one of flie addresses that belongs to the IPE card identified by the user's Primary Userld, 

^^^f-r thvi^ '^^^^*^-^^^^''^^^ based on the h ash of flic source and destination IP 

addresses of the packet, die LC must use the SPI in the IPsec header to select flie Userld, and flms flie 
IPE PPU, to send the packet to. In order to accomplidi this, the most significant 4 bits of an SPI 
always contain flic PPUID identifying flie FPU fliat is handling flie SA id^tified by fliat SPL 

1.3.2.3^ Outbound IPsec processing 

^ince IKccpcifonns tunneling at Layer 37cntire users don't get tunneled Rather, each packet about — 
to be sent to a user is individually examined using flie Security Policy Database (SPD) associated with 
that user, fiom which a pointer to a SA (in flie SAD also associated wxfli die user) is obtained 

The difficulty with outiwund processing is that, as discussed earlier, the configuration infoimation 
(and flins the SPD) associated with the egress user is not readily available. The infonnati<»i most be 
requested fiom flie PPU identified by the iVunoo'^^^ and stored in a cache. Each PPU sendii^ 
to a user vnSk thus create its own set of Security Associations. 



1«Zh7 IP£ Packet Fon/ifanting and Egress PMces^jg 

The IPE card FPUs performs routing table searches for aU packets fliat'need forwarding. The global 
Forwarding Information Base (FIB) is distributed to cv«y FPU in flie system, and contains IP unicast 
and multicast routing tables in a form fliat facilitates longest matodung prefix searches (Lc.: Patricia 
tries), as well as tables required for MPLS label based forwarding. 

One of flie results of every nniflng table lookup is the Primary Userld identifying flie layer 2 intrafiice 
by which flie packet should be transmitted It is inqiortant to note fliat flic iWwoiy (/^€r/(i is not fli^ 
same as flie LCUserld, and docs not direcdy give flic Qudid of flic Line Card where flic padret sfaodkl 
be forwarded Raflier, flie Primary Userld identifies flie IPE PPU flmt main t ains flie configqraflon and 
' state information for flie user. 

This presents a conqilicaflon because flie IPE fliat is trying to forward flie packet needs flie 
Mormation stored on flie IPE FPU identified by flie /Viwtao't/j^^^ Raflier flian simply forward flie 
packet to flie other IPE for egress processing, which would result in adctitional latency and switch 
fabric bandwidth utilization for every packet, it soids a message to ^e FPU identified by flie Primary 
Userld, requesting a copy of flic user 's configuration. This information is kqit in a user configuration 
cadie and is used for all subsequent packets directed to tiie same user. AU counters and statistics fliat 
need to be maintained for each usct must also be maintained for eadi cached user, and miust also be 
periodical^ sent to flic PPU identified by flie Primary Userld. 

Tins process makes it difBcuit to in^ilement such fimcticmality as per-uso- traffic shaping in flie IPE 
PPU, because the processing would need to be distributed among a potentially large number of 
processors. Therefore, traffic shaping is to be inq)!«ncntBd stricfly on flic Une Card uri^ 
FPUs. 

One of the fields that is acquired and cached as part of flic user configuration information is flie 
LCUserld. This field contains flie Ondld of flic Line C^d fliat flie packet must be for^^ 
weU as flie FFUm and cm fliat shooM be sent in flic Pm header of flie packet to fliat liiie 
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What is claimed is: 

1. A packet processing circuit comprising: 

a packet inspector for examining a stream of cells 
to de^tefmine control~iirf oriSati"on~f'o"r~~^^ 
thereby; 

5 at least one buffer access controller connected to 

said packet inspector for storing at least a portion of 
data cells received from said packet inspector, ^nd for 
processing control information received from said packet 
inspector to produce additional control information; and 
10 a packet manager connected to said buffer access 

controller for receiving control information therefrom 
for use in formatting packets corresponding to said 
control information. 

2. The circuit of claim 1 wherein said packet 
manager is configured for using the control information 
received from said buffer access controller to reassemble 
said corresponding packets . 

3. The circuit of claim 2 wherein said packet 
manager is connected to said packet inspector for 
coordinating the dequeuing of data cells representing 
said corresponding packets from said buffer access 

5 controller* 

4. The circuit of claim 1 further comprising a cell 
buffer associated with said buffer access controller for 
storing said data cells. 

5. The circuit of claim 4 further comprising at 
least one protocol processing unit associated with said 
buffer access controller for processing said control 
information received from said packet inspector. 

6. The circuit of claim 5 wherein said protocol 
processing unit comprises at least one general purpose 
processor unit. 
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7. The circuit of claim 5 further comprising an 
additional buffer access controller connected to said 
packet inspector, wherein said buffer access controllers . 
are configured for storing different pdrlTiTms^of "data 

5 cells received from said packet inspector. 

8. The circuit of claim 7 further comprising a 

_ _ prjotpcol processing unit associated with said additional 
buffer access controller, and wherein said buffer access 
controllers are each configured for determining whether 

5 to forward certain control information received from said 
packet inspector to its associated protocol processing 
unit for processing. 

9. The circuit of claim 8 further comprising a 
master processing unit connected to said protocol 
processing units for providing said protocol processing 
units with configuration data. 

10. The circuit of claim 9 further comprising a 
switch, wherein said master processing unit and said 
protocol processing units are interconnected to one 
another through said switch. 

11. The circuit of claim 7 wherein each buffer 
access controller has at least two protocol processing 
units associated therewith. 

12. A mid-network server comprising: 

an input for receiving a packet delivered thereto; ^ 
a line module connected to said input for receiving 
said packet; 

5 a plurality of processing modules for performing 

mid-network processing functions; and 

a switch fabric connected to said line module and 
said processing modules for delivering packets 
therebetween, wherein said processing modules are at 
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10 least substantially identical to one another and 
independently programmable • 

13. The server of claim 12 further comprising an 
additional line module, whereilT saTd~lT.^i(F are at 

least substantially identical to one another and 
independently programmable. 

14 - _ The^ server of claim 12 wherein said processing 

modules are each configured to support a plurality of 
packet types, and each line module is configured for 
formatting a packet into one of said types prior to 
5 sending said packet through said switch fabric to one of 
said processing modules. 

15. The server of claim 12 wherein said line module 
and said processing modules each comprise a packet 
inspector, a packet manager, and at least one buffer 
access controller. 

16. The server of claim 15 wherein said line module 
and said processing modules each comprise a plurality of 
buffer access controllers interconnected with said packet 
inspector and said packet manager. 

17. The server of claim 16 wherein each of said 
buffer access controllers have at least one protocol 
processing unit associated therewith. 

18. The server of claim 17 wherein each protocol 
processing unit is in communication with at least one 
other protocol processing unit on the same module. 

19. The server of claim 18 wherein said line module 
and said processing modules each comprise a master 
protocol processing unit for controlling the protocol 
processing units on that module. 

20. The server of claim 19 wherein said line module 
and said processing modules each comprise an Ethernet 
switch for interconnecting the master protocol processing 
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unit with said other protocol processing units on that 
5 module. 

21. A packet server comprising: 

an input for receiving a pal: ket^defivered^t hereto-; 

a line module connected to said input for receiving 
said packet; 

5 a plurality of processing modules for performing 

packet routing functions; and ^ 7" 

a switch fabric connected to said line module and 
said processing modules for delivering packets 
therebetween, wherein said line module is configurable to 

10 send said packet to any one of said processing modules 
through said switch fabric, and said processing modules 
are each configurable to perfomn said routing functions 
for said packet if said packet is sent thereto by said 
line module - 

22- The server of claim 21 wherein said line module 
supports a plurality of user interfaces and is configured 
to send said packet to one of said processing modules 
according to the user interface through which said packet 
5 arrives at said server, 

23. The server of claim 22 wherein each processing 
module includes a plurality of processing units, and said 
line module is configured to send said packet to one of 
said processing units of one of said processing modules 

5 according to the user interface through which said packet 
arrives at said server. 

24. The server of claim 23 wherein said processing 
modules are each configured to support a plurality of 
packet types, and said line module is configured for 
formatting said packet into one of said types prior to 

5 sending said packet through said switch fabric to one of 
said processing modules. 
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25. The server of claim 21 wherein said line module 
is connected to said input through a phy module. 

26. The server of claim 21 wherein said line modul-e 

an'd*^sai"d~processing-moduies— each-~±nclude~a--pl-ura-l^^ 

general purpose processing units. 

27. The server of claim 21 wherein said line module 
and said processing modules can be programmed to support 
any type of transmission protocol. ~ 

28. A packet server comprising: 

a plurality of line modules for receiving packets 
delivered to said server over a physical connection; 

at least one processing module for performing packet 
5 routing functions; and 

a switch fabric connected to said line modules and 
said processing module for delivering packets 
therebetween, wherein each line module is configured to 
format a packet into one of a plurality of types prior to 
10 sending said packet through said switch fabric to said 
processing module, and said processing module is 
configured to support each of said packet types . 

29. The server of claim 28 wherein said processing 
module includes a plurality of processing units . 

30. The server of claim 29 wherein each line card 
supports a plurality of users and is configured to assign 
each user to one of the processing units of said 
processing module. 

31. The server of claim 30 wherein at least one of 
the processing units of said processing module is 
assigned to a first user supported by a first one of said 
line modules and a. second user supported by a second one 

5 of said line modules. 

32. A method for processing packets within a 
server, said method comprising the steps of: 
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converting a packet input to said server into a 
stream of fixed length cells; 
5 processing said stream of fixed length cells using a 

Iilie"lno3ule~t^~fofm^ 
of protocol types; and 

sending said formatted packet to a processing module 
configured to support each of said plurality of protocol 
10 types . 

33. The method of claim 32 wherein said processing 
step includes reassembling said packet. 

34. The method of claim 33 wherein said processing 
step further includes examining said cell stream to 
obtain control information for said packet. 

35. The method of claim 34 wherein said control 
information includes infonnation identifying a particular 
processing module for further processing said packet. 

36. The method of claim 34 wherein said processing 
step further includes processing said control information 
to produce additional control information for use in 
reassembling and formatting said packet. 

37. The method of claim 36 wherein said processing 
step further includes identifying, a particular processing 
module to which said packet should be sent. 

38. The method of claim 37 wherein the sending step 
includes sending said packet to said particular 
processing module identified in said processing step. 

39. The method of claim 37 wherein said identifying 
step includes identifying a particular protocol 
processing unit on said particular processing module for 
processing control information corresponding to said 

5 packet. 
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40. The method of claim 33 further comprising the 
step of performing mid-network processing functions on 
said sent packet using said processing module. 

4-1- ^rhe-method-of-Glai-m-40-wherein— the-step-of — 

performing mid-network processing functions includes 
formatting said packet for its destination interface. 

42. The method of claim 41 further comprising the 
step of "sending" the packet" fbrmafted by" sa~id 'processing 
module to a line module corresponding to said destination 
interface. 

43. The method of claim 32 wherein said sending 
step includes sending said packet through a switch 
fabric. 

44. The method of claim 32 wherein said input 
packet is a packet represented by a plurality of fixed 
length cells. 

45. The method of claim 4 4 wherein the converting 
step includes modifying the length of said input cells. 

46. A method for processing packets within a 
server, said method comprising the steps of: 

converting a packet input to said server into a 
stream of fixed length cells; 
5 processing said stream of fixed length cells using a 

line module to format said packet into one of a' plurality 
of protocol types; and 

sending said formatted packet to another line module 
configured to support each of said plurality of protocol 
10 types. 

47. A method for processing packets within a 
server, said method comprising the steps of: 

converting a packet input to said server into a 
stream of fixed length cells; 
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5 processing said stream of fixed length cells using a 

line module to format said packet into one of a plurality 
of protocol types; 

sending~said-"formatted-packet"-to--a-processi-ng-modu-l^^ 

configured to support each of said plurality of protocol 
10 types; and 

processing said stream of fixed length cells in said 
processing module; 

reformatting said formatted packet into one of a 
plurality of protocol types; and 
15 sending said reformatted packet to another 

processing module. 
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